How to Read Large File in Java

In our last article, we cover How to read file in Java.This post will cover how to read large file in Java efficiently.

Reading the large file in Java efficiently is always a challenge, with new enhancements coming to Java IO package, it is becoming more and more efficient.

We have used sample file with size 1GB for all these. Reading such a large file in memory is not a good option, we will covering various methods outlining How to read large file in Java line by line.

1 Using Java API

We will cover various options how to read a file in Java efficiently using plain Java API.

1.1 Using Java BufferReader


public class ReadLargeFileByBufferReader {

    public static void main(String[] args) throws IOException {

        String fileName = "/tutorials/fileread/file.txt"; //this path is on my local
        try (BufferedReader fileBufferReader = new BufferedReader(new FileReader(fileName))) {
            String fileLineContent;
            while ((fileLineContent = fileBufferReader.readLine()) != null) {
                // process the line.
            }
        }
    }
}

Output


Max Memory Used : 258MB
Time Take : 100 Seconds

1.2 Using Java 8 Stream API

public class ReadLargeFIleUsingStream {

    public static void main(String[] args) throws IOException {

        String fileName = "/tutorials/fileread/file.txt"; //this path is on my local
        // lines(Path path, Charset cs)
        try (Stream inputStream = Files.lines(Paths.get(fileName), StandardCharsets.UTF8)) {
            inputStream.forEach(System.out::println);
        }
    }
}

Output

Max Memory Used : 390MB
Time Take : 60 Seconds

1.3 Using Java Scanner

Java Scanner API also provides a way to read large file line by line.


  public class ReadLargeFileByScanner {

    public static void main(String[] args) throws FileNotFoundException {
        String fileName = "/Users/umesh/personal/tutorials/fileread/file.txt"; //this path is on my local
        InputStream inputStream = new FileInputStream(fileName);
       
        try(Scanner fileScanner = new Scanner(inputStream, StandardCharsets.UTF_8.name())){
            while (fileScanner.hasNextLine()){
                System.out.println(fileScanner.nextLine());
            }
        }
    }
}

Output


Max Memory Used : 460MB
Time Take : 60 Seconds

2 Streaming File Using Apache Commons IO

This can also be achieved by using Apache Commons IO FileUtils.lineIterator () Method


public class ReadLargeFileUsingApacheCommonIO {

    public static void main(String[] args) throws IOException {
        String fileName = "/Users/umesh/personal/tutorials/fileread/file.txt"; //this path is on my local

        LineIterator fileContents= FileUtils.lineIterator(new File(fileName), StandardCharsets.UTF_8.name());
        while(fileContents.hasNext()){
            System.out.println(fileContents.nextLine());   
        }
    }
}

Output


Max Memory Used : 400MB
Time Take : 60 Seconds

As we saw how to read a large file in Java efficiently. Few things which you need to pay close attention

Reading the large file in one go will not be a good option (You will get OutOfMemoryError ).
We Adapted technique to read large file line by line to keep memory footprint low.

I used VisualVM to monitoring Memory, CPU and Threadpool information while running these programmes.

based on our test, BufferReader has the lowest memory footprint, though the overall execution was slow.

All the code of this article is available Over on Github. This is a Maven-based project.

References

Apache Commons IO