JAVA / FILES, INPUT/OUTPUT AND SERIALIZATION
CSV and line-oriented data without a library
Parse and write CSV by hand in Java: a one-bit state machine for quoted fields, doubled quotes for escaping, and records that span lines.
What you will learn
- Split a CSV record with a one-bit state machine so quoted commas stay inside their field
- Escape output by wrapping the value in quotes and doubling every interior quote
- Use split(",", -1) so empty trailing columns are not dropped
- Buffer physical lines into one record while its quotes are unbalanced
Understanding CSV and line-oriented data without a library
A CSV file is not a table stored as text; it is an encoding in which the delimiter carries two different meanings. Outside a quoted region a comma ends a field, inside one it is ordinary data, and resolving that ambiguity is the entire job of the double quote. String.split decides at each position independently, with no memory of what came before, so it is only correct for data you have guaranteed contains no delimiter, quote or line break. Treat split as a shortcut that encodes an assumption about your data, and a parser as the thing that actually decodes the format.
The decoder needs precisely one bit of state: are we inside quotes right now? Outside, a quote opens a quoted region, a comma flushes the field being built, anything else is appended. Inside, every character is appended verbatim until a quote appears, and that quote is ambiguous on its own, so the parser peeks at the next character: another quote means one literal quote belongs in the field and both characters are consumed, anything else means the region just closed. That single lookahead is the whole reason CSV can carry commas, newlines and quotes without ever needing an escape character.
Writing is the mirror image, and it is where most corruption starts, because the writer is what decides whether a value needs quoting. Quote whenever the value contains the delimiter, a double quote, CR or LF, double the interior quotes, and leave everything else bare. Because a quoted field may hold a line break, one physical line is not one record, so a general reader has to keep pulling lines until the record's quotes balance. Nothing in the format marks the header line either; that is a convention your code applies, usually by turning the first record into a name-to-index map.
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class CsvRecord {
// One CSV record -> its fields. A comma inside a quoted region is data.
static List<String> parseRecord(String raw) {
List<String> fields = new ArrayList<>();
StringBuilder field = new StringBuilder();
boolean inQuotes = false;
for (int i = 0; i < raw.length(); i++) {
char c = raw.charAt(i);
if (inQuotes) {
if (c != '"') {
field.append(c);
} else if (i + 1 < raw.length() && raw.charAt(i + 1) == '"') {
field.append('"'); // "" stands for one literal quote
i++;
} else {
inQuotes = false; // closing quote
}
} else if (c == '"') {
inQuotes = true;
} else if (c == ',') {
fields.add(field.toString());
field.setLength(0);
} else {
field.append(c);
}
}
fields.add(field.toString()); // the last field has no comma after it
return fields;
}
static void show(String label, List<String> fields) {
System.out.println(label + ": " + fields.size() + " fields");
for (String f : fields) {
System.out.println(" [" + f + "]");
}
}
public static void main(String[] args) {
String row = "7,\"Doe, Jane\",\"5\"\" bolt\",,42";
System.out.println(row);
show("split", Arrays.asList(row.split(",", -1)));
show("parseRecord", parseRecord(row));
}
}In CSV the delimiter separates fields only outside a quoted region, so correct parsing needs one bit of state rather than a split.
Worked examples
Quoting on the way out
Builds a record from raw values, quoting only the ones whose content would otherwise change the record's shape.
public class CsvWrite {
static String encodeField(String value) {
boolean needsQuotes = value.indexOf(',') >= 0 || value.indexOf('"') >= 0
|| value.indexOf('\n') >= 0 || value.indexOf('\r') >= 0;
if (!needsQuotes) {
return value;
}
return '"' + value.replace("\"", "\"\"") + '"';
}
static String encodeRow(String... values) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < values.length; i++) {
if (i > 0) {
sb.append(',');
}
sb.append(encodeField(values[i]));
}
return sb.toString();
}
public static void main(String[] args) {
String row = encodeRow("plain", "Doe, Jane", "5\" bolt", "two\nlines", "");
System.out.println(row);
System.out.println("--- same text, newline made visible ---");
System.out.println(row.replace("\n", "\\n"));
}
}Example explained
Line 1needsQuotes tests exactly the four characters that would otherwise change the record's structure: the delimiter, the quote, CR and LF.
Line 2replace("\"", "\"\"") doubles every interior quote, because CSV has no backslash escape and repetition is the only escape it defines.
Line 3The empty last value stays bare, so the row ends with a comma; a reader calling split(",") without -1 would report four fields instead of five.
Line 4The fourth value holds a real newline, so this single record occupies two physical lines in the file.
Two ways split lies to you
Shows that split discards trailing empty fields and that its argument is a regular expression, not a literal delimiter.
import java.util.Arrays;
import java.util.regex.Pattern;
public class SplitTraps {
public static void main(String[] args) {
String row = "a,b,,";
System.out.println("default : " + Arrays.toString(row.split(","))
+ " len=" + row.split(",").length);
System.out.println("limit -1: " + Arrays.toString(row.split(",", -1))
+ " len=" + row.split(",", -1).length);
String piped = "a|b|c";
System.out.println("pipe : " + Arrays.toString(piped.split("|"))
+ " len=" + piped.split("|").length);
System.out.println("quoted : " + Arrays.toString(piped.split(Pattern.quote("|")))
+ " len=" + piped.split(Pattern.quote("|")).length);
}
}Example explained
Line 1"a,b,," is a four-column row, but with the default limit of 0 split strips the trailing empty strings and hands back two elements.
Line 2Passing -1 keeps every empty field, which is the only form safe to use when you compare against an expected column count.
Line 3split("|") compiles | as alternation between two empty branches, so it matches at every position and yields one element per character, pipes included.
Line 4Pattern.quote("|") wraps the delimiter in \Q...\E so it is matched as a literal character.
One record, several lines
Joins physical lines until the accumulated record has an even number of quote characters, so a newline inside a quoted field survives.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.StringReader;
public class MultiLineRecords {
static boolean quotesBalanced(CharSequence s) {
int quotes = 0;
for (int i = 0; i < s.length(); i++) {
if (s.charAt(i) == '"') {
quotes++;
}
}
return quotes % 2 == 0;
}
public static void main(String[] args) throws IOException {
String csv = "id,note\n"
+ "1,\"first line\nsecond line\"\n"
+ "2,plain\n";
System.out.println("physical lines: " + csv.split("\n").length);
BufferedReader in = new BufferedReader(new StringReader(csv));
StringBuilder buffer = new StringBuilder();
String line;
int records = 0;
while ((line = in.readLine()) != null) {
if (buffer.length() > 0) {
buffer.append('\n'); // this newline was field content
}
buffer.append(line);
if (quotesBalanced(buffer)) {
records++;
System.out.println("record " + records + ": "
+ buffer.toString().replace("\n", "\\n"));
buffer.setLength(0);
}
}
System.out.println("records: " + records);
}
}Example explained
Line 1The document has four lines but three records, because the note field carries a newline between its quotes.
Line 2readLine() drops the line terminator, so '\n' is appended back before the continuation, otherwise the two halves of the field would be glued together.
Line 3An even quote count means no field is left open; a doubled "" adds two quotes, so the parity test still holds for escaped quotes in well-formed data.
Line 4Only after the buffer balances is the text handed to a field parser, so record boundaries and field boundaries stay separate concerns.
Important notes
The doubled-quote rule applies only inside a quoted field. A bare quote in an unquoted field, such as 5" bolt written without quotes, is not valid CSV and producers disagree about it, so quote such values when you write them.
A UTF-8 byte order mark at the start of the file arrives as a leading '\uFEFF' on the first header name, so "id".equals(header.get(0)) fails even though the text looks identical; strip it before matching headers.
Common mistakes
Calling line.split(",") on data with quoted commas: the row breaks into too many fields, and every column after the quoted one is silently shifted by one.
Omitting the -1 limit: a row ending in an empty column comes back short, so reading the last column throws ArrayIndexOutOfBoundsException or the row is rejected as malformed.
Splitting whole-file text on "\n" when the file uses CRLF: the last field of every row keeps a trailing '\r', so Integer.parseInt("42\r") throws NumberFormatException and equals comparisons fail.
Try it yourself
Change, predict, then run
Copy parseRecord from the lesson and replace the hardcoded comma with a char delimiter parameter, then parse x;"a;b";;c with ';' and print each field in brackets. You should see four fields, with a;b as the second and an empty third.
Open the Java workspaceCheck your understanding
You control both the writer and the reader of a data file. Which single guarantee about the data makes line.split(",", -1) a correct CSV parser?
- Every row has the same number of columns
- No field ever contains a comma, a double quote, a CR or an LF
- Every field is wrapped in double quotes
- The file is UTF-8 with CRLF line endings and no byte order mark
Show answer
split has no notion of quoting; it cuts at every delimiter it finds, wherever it finds it, so its correctness is a property of the data rather than of the code. If no value can hold a comma, a quote or a line break, there is nothing to quote and nothing to escape, and splitting is exact. Wrapping every field in quotes does not help, because split still cuts inside "Doe, Jane" and leaves you with the fragments "Doe and Jane" to reassemble; equal column counts and the file encoding matter for other reasons but do nothing to make a delimiter inside a value safe.