Fastest way to read files in Java

Introduction:

In the Universal Media Server project, we recently ran some benchmarks to discover the fastest way to read files, particularly big files like HD movies. We tested four methods using an automatic benchmark script:

  1. FileChannel using File input
  2. FileChannel using Path input
  3. DataInputStream using File input
  4. RandomAccessFile using File input

We tested these on different hard drives with different rotation speeds, and with files from 600MB up to 22GB each, and using 1-100 threads to see what effect that had on the results.

Results:

We experienced different results but on average for our use case, we found that the two FileChannel methods were the best, and went with the second option since the Path input is the newer syntax in Java. The DataInputStream and RandomAccessFile had significantly slow outliers that had been causing problems on some hard drives.

My results:

FileChannel using File input:
Benchmarking of hashing 152000 files using 1 thread took 57277 ms (376824 ns average per file)
Benchmarking of hashing 152000 files using 100 threads took 20130 ms (132437 ns average per file)

FileChannel using Path input:
Benchmarking of hashing 152000 files using 1 thread took 56675 ms (372867 ns average per file)
Benchmarking of hashing 152000 files using 100 threads took 21373 ms (140615 ns average per file)

DataInputStream using File input:
Benchmarking of hashing 152000 files using 1 thread took 75716 ms (498133 ns average per file)
Benchmarking of hashing 152000 files using 100 threads took 330825 ms (2176486 ns average per file)

RandomAccessFile using File input:
Benchmarking of hashing 152000 files using 1 thread took 51090 ms (336121 ns average per file)
Benchmarking of hashing 152000 files using 100 threads took 326446 ms (2147671 ns average per file)

For other results and more details, check out the branch with the benchmarking code

Also note that we were doing a specific type of hashing that is used by OpenSubtitles, which involves reading the beginning and end of the file, so other uses of the reads may give different results.

Universal Media Server 7.0.0 released


This release adds new features to dynamically sort and add your media, most notably there are new “TV Shows” and “Movies” folders that are populated automatically with content from your shared folders.

Download

Release details

Universal Media Server 7.0.0-b1 released


The 7.0.0 version adds new features to dynamically sort your media, most notably there are new “TV Shows” and “Movies” folders that are populated automatically with content from your shared folders.

Release details and download

Universal Media Server 6.5.0 released


We have added support for LG Blu-ray players and Naim Mu-So wireless audio systems, improved support for Panasonic and Samsung TVs and the VLC app, fixed videos being transcoded too often, fixed a bunch of bugs, and more!

Download

Release details

Universal Media Server 6.3.0 released


This release adds and improves support for Yamaha, Miracast and Panasonic devices, 3D and 2D subtitles, improves transcoding quality and speed, squashes bugs, and more!

Download

Release details

MPlayer & MEncoder SB65 released

MPlayer logo
For more details, please see the main post.

Changes since SB64:

  • Updated x264 to r2695, which improves speed and quality

Universal Media Server 6.0.0-b1 released


This release adds actions when media has been fully played and improves accuracy of timers on the Status tab.

Download

Release details

Universal Media Server 5.4.0 released


This release adds support for AnyCast, Miracast M806 and more Sony Bravia TVs, improves speed, file support, renderer support, bugfixes and more!

Download

Release details

MPlayer & MEncoder SB64 released

MPlayer logo
For more details, please see the main post.

Changes since SB63:

  • Updated x264 to r2638
  • Updated fontconfig to 2.11.92

Universal Media Server 5.2.2 released


This release improves file support, speed, translations, renderer support and more!

Download

Release details

© 2018 Spirton

Theme by Anders NorenUp ↑