JAVA / CAPSTONE PROJECTS
Project: a stream-powered log analyser with a summary report
Build a log analyser that parses lines into records, aggregates them with groupingBy and teeing, and prints a reproducible summary report.
What you will learn
- Parse each line into a record and drop junk with flatMap(Optional::stream)
- Build report sections with groupingBy plus counting or summarizingInt downstream
- Compute two unrelated aggregates in a single traversal with Collectors.teeing
- Keep report rows stable with TreeMap::new and format numbers with an explicit Locale
Understanding Project: a stream-powered log analyser with a summary report
A log analyser has three stages and it pays to keep them apart: text in, records in the middle, numbers out. The parse step is the only code that knows about spaces and column positions, and having it return Optional<Entry> means a truncated or half-rotated line simply disappears at flatMap(Optional::stream) instead of throwing an exception two thirds of the way through the file. Once the stream carries Entry objects rather than strings, every question you ask of the log (how many warnings, which endpoint is slowest) becomes a collector instead of a loop, because you name the shape of the answer and let the library do the accumulating.
A stream is a one-shot pipeline, not a collection: the first terminal operation marks it consumed, and a stream from Files.lines also holds an open file descriptor until you close it. That forces a decision as soon as the report needs more than one figure. For a report-sized log, collecting into a List once and then running several small pipelines over that list is the readable choice; when the input is too large to hold in memory, compute the figures together with groupingBy downstreams or Collectors.teeing so the lines are read exactly once.
Most of a summary report is one groupingBy away. counting() as the downstream collector turns a classifier into a frequency table, and summarizingInt(Entry::millis) returns count, min, max, sum and average in a single object, which is why per-endpoint latency does not need four separate passes. Two details make the output trustworthy: groupingBy returns a HashMap whose iteration order is unspecified, so pass TreeMap::new or sort the entry set with an explicit tie-break, and format numbers with an explicit Locale, otherwise the same log can print its rows in a different order or with a comma decimal separator on someone else's machine.
import java.time.LocalTime;
import java.util.*;
import java.util.stream.Collectors;
public class LogAnalyser {
record Entry(LocalTime time, String level, String endpoint, int millis) {}
private static final String LOG = """
09:14:02 INFO /api/users 120
09:14:03 WARN /api/users 480
09:14:07 INFO /api/orders 95
09:14:09 ERROR /api/orders 1500
09:14:11 INFO /api/users 60
09:14:15 ERROR /api/payments 2200
09:14:16 INFO /api/orders 130
""";
private static Optional<Entry> parse(String line) {
String[] f = line.trim().split("\\s+");
if (f.length != 4) return Optional.empty();
try {
return Optional.of(new Entry(LocalTime.parse(f[0]), f[1], f[2], Integer.parseInt(f[3])));
} catch (RuntimeException malformed) {
return Optional.empty();
}
}
public static void main(String[] args) {
List<Entry> entries = LOG.lines()
.map(LogAnalyser::parse)
.flatMap(Optional::stream)
.toList();
Map<String, Long> perLevel = entries.stream().collect(
Collectors.groupingBy(Entry::level, TreeMap::new, Collectors.counting()));
Map<String, IntSummaryStatistics> perEndpoint = entries.stream().collect(
Collectors.groupingBy(Entry::endpoint, TreeMap::new,
Collectors.summarizingInt(Entry::millis)));
System.out.println("parsed entries: " + entries.size());
System.out.println("per level: " + perLevel);
perEndpoint.forEach((endpoint, s) -> System.out.printf(Locale.ROOT,
"%-14s n=%d avg=%.1fms max=%dms%n",
endpoint, s.getCount(), s.getAverage(), s.getMax()));
System.out.println("slowest: " + entries.stream()
.max(Comparator.comparingInt(Entry::millis))
.map(e -> e.endpoint() + " " + e.millis() + "ms at " + e.time())
.orElse("no data"));
}
}Aggregate parsed records with collectors rather than loops over strings, and design the report around how few times you can traverse the log.
Worked examples
Two figures, one traversal
Collectors.teeing produces an error count and an average latency from a single pass over the entries.
import java.util.List;
import java.util.Locale;
import java.util.stream.Collectors;
public class OnePassReport {
record Entry(String level, String endpoint, int millis) {}
record Summary(long errors, double avgMillis) {}
public static void main(String[] args) {
List<Entry> entries = List.of(
new Entry("INFO", "/api/users", 120),
new Entry("ERROR", "/api/orders", 1500),
new Entry("INFO", "/api/orders", 95),
new Entry("ERROR", "/api/payments", 2200));
Summary summary = entries.stream().collect(Collectors.teeing(
Collectors.filtering(e -> e.level().equals("ERROR"), Collectors.counting()),
Collectors.averagingInt(Entry::millis),
(errors, avg) -> new Summary(errors, avg)));
System.out.println(summary);
System.out.printf(Locale.ROOT, "error rate %.0f%% over %d entries%n",
100.0 * summary.errors() / entries.size(), entries.size());
}
}Example explained
Line 1teeing hands every entry to both downstream collectors as it goes past, then merges the two results once at the end.
Line 2Collectors.filtering restricts the count to ERROR lines inside that branch only; a .filter() before collect would have removed those lines from the average as well.
Line 3counting() yields Long and averagingInt yields Double, so the merge function receives boxed values that the Summary constructor unboxes.
Line 4Returning a record instead of printing means the aggregate can be passed to a formatter or asserted on in a test.
Top endpoints with a deterministic order
Turns request paths into a frequency table and ranks it with an explicit tie-break so the report never shuffles.
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
public class TopEndpoints {
public static void main(String[] args) {
List<String> requests = List.of(
"/api/users", "/api/orders", "/api/users", "/api/payments",
"/api/orders", "/api/users", "/api/health", "/api/orders");
Map<String, Long> hits = requests.stream()
.collect(Collectors.groupingBy(path -> path, Collectors.counting()));
Comparator<Map.Entry<String, Long>> byHits = Map.Entry.comparingByValue();
Comparator<Map.Entry<String, Long>> byPath = Map.Entry.comparingByKey();
String report = hits.entrySet().stream()
.sorted(byHits.reversed().thenComparing(byPath))
.limit(3)
.map(e -> " " + e.getKey() + " -> " + e.getValue())
.collect(Collectors.joining("\n", "top 3 endpoints\n", "\n"));
System.out.print(report);
}
}Example explained
Line 1groupingBy with an identity classifier and counting() collapses eight request lines into four keys, which is all a frequency table is.
Line 2counting() produces Long values, so comparingByValue() compares numbers; comparing the formatted strings instead would put 10 before 9.
Line 3sorted() is stable, so without the secondary comparator on the key the two three-hit endpoints would appear in HashMap layout order, which is not guaranteed.
Line 4joining with a prefix and suffix assembles the whole section as one string, so nothing reaches the console until the pipeline finishes.
Reading the real file once
Shows that a Files.lines stream must be closed and cannot be traversed a second time.
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;
public class LogFileStream {
public static void main(String[] args) throws IOException {
Path log = Files.createTempFile("app", ".log");
Files.writeString(log, """
INFO /api/users 120
#### rotated
ERROR /api/orders 1500
""");
try (Stream<String> lines = Files.lines(log)) {
long wellFormed = lines.filter(line -> line.split(" ").length == 3).count();
System.out.println("well formed lines: " + wellFormed);
try {
lines.count();
} catch (IllegalStateException reused) {
System.out.println("second pass: " + reused.getMessage());
}
}
Files.delete(log);
}
}Example explained
Line 1Files.lines is lazy: bytes are read only as count() pulls elements, which is why the file handle stays open for the whole block.
Line 2The first terminal operation flags the pipeline as consumed, so the second count() throws before reading anything rather than returning 0.
Line 3Producing several figures from one file therefore means reopening it or aggregating in one pass with groupingBy or teeing.
Line 4Files.delete succeeds only because try-with-resources closed the stream first; an unclosed stream keeps the log locked on Windows.
Important notes
IntSummaryStatistics for an empty group reports an average of 0.0 and a max of Integer.MIN_VALUE, so check that entries exist before printing latency numbers.
Files.lines decodes as UTF-8 and throws UncheckedIOException part-way through the pipeline on invalid bytes; pass an explicit charset such as ISO_8859_1 for older logs.
Common mistakes
Obtaining the stream from Files.lines outside try-with-resources: the descriptor stays open until garbage collection, and the log cannot be rotated or deleted while it is held.
Incrementing a shared field inside forEach or peek instead of returning a collector result: the totals are silently wrong as soon as the stream is parallel, and the figure cannot be reused by a caller.
Assuming groupingBy hands back rows in log order: it returns a HashMap, so report lines appear in an unspecified order unless you pass TreeMap::new or sort the entry set yourself.
Try it yourself
Change, predict, then run
Add an errors-per-minute section to the analyser: keep only entries with level ERROR, group them by entry.time().truncatedTo(ChronoUnit.MINUTES) with counting(), and print every minute that has two or more errors.
Open the Java workspaceCheck your understanding
Your analyser opens a log with Files.lines(path), calls count() to report the number of lines, then calls filter(...).count() on the same Stream variable to count errors. Why does the second call fail even though the file is still open?
- count() closed the underlying file, so there is nothing left to read
- The file's read position is already at the end and Files.lines cannot rewind, so the second count returns 0
- A stream is a single-use pipeline: the first terminal operation marks it consumed, so any later operation throws IllegalStateException
- filter() may not be added after count() because counting removes the SIZED characteristic from the pipeline
Show answer
The pipeline records that it has been operated upon, so the next intermediate or terminal call throws IllegalStateException before a single byte is read again; the fix is to reopen the file or compute both figures in one pass with teeing. The rewind answer is tempting because the file cursor really is at the end, but the failure is an exception raised by the stream rather than a quiet 0, and count() does not close anything either, which is exactly why the try-with-resources block is needed.