C / STRINGS AND BUFFER SAFETY
Tokenising and parsing strings with strtok carefully
Split a writable C string into fields with strtok, predict which tokens it silently drops, and know when to reach for strtok_r instead.
What you will learn
- Drive the strtok(buf, d) / strtok(NULL, d) loop and stop when it returns NULL
- Predict that runs of delimiters collapse, so strtok never returns an empty field
- Switch to strtok_r with your own saveptr for nested or library-internal tokenising
- Explain why strtok on a string literal is an illegal write to read-only memory
Understanding Tokenising and parsing strings with strtok carefully
strtok does not split a string into new strings; it edits the one you handed it. Each call scans forward from a position it remembered privately, overwrites the first delimiter it finds with a NUL byte, and returns a pointer into your own buffer just before that byte. That is why the first argument must be modifiable memory that stays alive as long as you hold the token pointers, and why the line you tokenised is afterwards no longer the line you started with.
The second argument is a set of individual characters, not a separator sequence, and strtok skips any run of those characters before it starts a token. The consequence people meet the hard way is that it can never hand back an empty token: splitting 0:0::/root on a colon yields three tokens, not four, and every field after the gap is numbered one too low. strtok is a word splitter, good for chopping a command into arguments or a sentence into words, and the wrong tool for record formats where a field may legitimately be empty.
The remembered position lives in a single static slot inside the library, one per thread in practice, not one slot per string, so only one tokenisation can be in flight at a time. Nesting two loops, or calling any function that itself uses strtok from inside a strtok loop, quietly hijacks the outer cursor: the outer loop resumes inside the inner string and then ends when that string runs out. strtok_r takes a char **saveptr that you own, which is exactly what makes it safe in nested loops, in reusable functions, and in threads.
<stdio.h>
<string.h>
int main(void)
{
char line[] = "root:x:0:0:root:/root:/bin/sh";
char *field;
int n = 0;
for (field = strtok(line, ":"); field != NULL; field = strtok(NULL, ":")) {
printf("field %d = [%s]\n", ++n, field);
}
/* every ':' strtok consumed is now a NUL byte inside line */
printf("line[4] is now %d, line[5] is still '%c'\n", line[4], line[5]);
return 0;
}
strtok is a destructive cursor: it writes NUL bytes into your buffer, returns pointers into that same buffer, and keeps its one saved position in hidden shared state.
Worked examples
Empty fields disappear
Shows that a run of delimiters produces no token at all, which breaks comma-separated records.
<stdio.h>
<string.h>
int main(void)
{
char csv[] = "10,,30,,";
char *tok;
int n = 0;
for (tok = strtok(csv, ","); tok != NULL; tok = strtok(NULL, ",")) {
printf("token %d = [%s]\n", ++n, tok);
}
printf("got %d tokens from a line holding 5 comma-separated fields\n", n);
return 0;
}
Example explained
Line 1Splitting 10,,30,, at every comma gives five fields, three of which are empty.
Line 2strtok skips the whole run of two commas before starting a token, so the gap between 10 and 30 never becomes a token.
Line 3The two trailing commas leave only delimiters, so the third call returns NULL and the loop ends.
Line 4Field-oriented data therefore needs a hand-written strchr scan or strsep, which do report empty fields.
Nested splitting with strtok_r
Splits pairs on semicolons and each pair on an equals sign at the same time, using two independent saved positions.
_POSIX_C_SOURCE
<stdio.h>
<string.h>
int main(void)
{
char cfg[] = "host=db1;port=5432;tls=on";
char *outer, *inner, *pair, *key, *val;
for (pair = strtok_r(cfg, ";", &outer); pair != NULL; pair = strtok_r(NULL, ";", &outer)) {
key = strtok_r(pair, "=", &inner);
val = strtok_r(NULL, "=", &inner);
printf("%-5s -> %s\n", key, val != NULL ? val : "(missing)");
}
return 0;
}
Example explained
Line 1outer holds the position of the semicolon scan and inner holds the position of the equals scan, so neither loop can disturb the other.
Line 2The inner call passes pair rather than NULL, which starts a fresh tokenisation of that one field.
Line 3_POSIX_C_SOURCE is defined before the headers because strtok_r comes from POSIX, not from ISO C.
Line 4key and val point into cfg itself, so this whole parse allocates nothing.
A different delimiter on every call
Demonstrates that the delimiter set belongs to the individual call, not to the tokenisation as a whole.
<stdio.h>
<string.h>
int main(void)
{
char stamp[] = "2026-09-03 20:01:16";
char *y = strtok(stamp, "-");
char *mo = strtok(NULL, "-");
char *d = strtok(NULL, " ");
char *h = strtok(NULL, ":");
char *mi = strtok(NULL, ":");
char *s = strtok(NULL, ":");
if (!y || !mo || !d || !h || !mi || !s) {
fputs("malformed timestamp\n", stderr);
return 1;
}
printf("%s/%s/%s %sh%sm%ss\n", d, mo, y, h, mi, s);
return 0;
}
Example explained
Line 1The same hidden cursor walks the buffer while the delimiter set changes from - to space to :, because each call names its own set.
Line 2Initialisers in a block run in order, so the six calls happen top to bottom as written.
Line 3The final call finds no colon left, so the token simply runs to the end of the buffer.
Line 4The NULL check matters because a malformed input makes one call fail and every later call then also returns NULL, which would feed %s a null pointer.
Important notes
The delimiter argument is a set of characters, so strtok(s, ", \t") breaks on a comma, a space or a tab in any order and any quantity; there is no way to say the separator is exactly one comma.
strtok(line, "\n") is a common way to trim the newline left by fgets, but on a line that is only a newline it returns NULL rather than an empty string.
Common mistakes
Passing a literal, as in char *s = "a,b"; strtok(s, ","); strtok tries to store a NUL over the comma in read-only memory and the program dies with SIGSEGV, with nothing wrong-looking at the call site.
Expecting an empty string for a missing field: on root:x:0:0::/root:/bin/sh the double colon collapses, so what you count as field 6 is the shell and not the home directory, and nothing reports an error.
Reusing the line after tokenising it: the delimiters are gone, so printf("%s", line) prints only the first token and strlen(line) returns that token's length.
Try it yourself
Change, predict, then run
Tokenise char rec[] = "alice,,30,london"; on commas, print each token in brackets, then print rec on its own line and explain why only part of it survives. Rewrite the same split using strchr so that the empty second field prints as [].
Open the C workspaceCheck your understanding
You loop over the fields of a line with strtok, and inside the loop body you call a helper function that itself uses strtok on a completely different string. What is the most likely result?
- Nothing changes, because strtok keeps a separate saved position for each string it has been given.
- The outer loop resumes inside the helper's string and then ends early, because both calls share one saved position.
- The helper's first call returns NULL, because strtok refuses to start a second tokenisation while one is active.
- Both loops work, but the helper's tokens are cut using the outer loop's delimiter set.
Show answer
strtok stores exactly one saved position and it is not associated with the string that was passed in, so the helper's strtok(other, ...) overwrites it; the outer strtok(NULL, ...) then continues inside the helper's buffer and returns NULL as soon as that buffer is exhausted. The first option is tempting because strtok appears to remember your string, but all it remembers is a pointer to where it stopped, which is why giving each loop its own saveptr through strtok_r is the actual fix.