fd twice as slow as rg for same dir traversals

[stderr-to-devnull]

fd version: 10.4.2

I ran the same file types listings on "cold" OS with buffers flushed:

• rg took 37s (second run after some time took 19s, with piping to lscolors)
• fd took 1m8s

Counted the returned results, 33679 for rg and 33676 for fd. So difference is inconsequential.

So it is quite strange that a program made to find strings inside files is much faster that a program made specifically to list fast dirs and files.

So I will use rg to find my files and pipe it to lscolors to achieve the same highlighting as fd. But it is quite confusing from a user perspective due to the expectation that fd would be the fastest tool possible to do this.

[tmccombs]

This isn't really actionable without more details on how to reproduce

[stderr-to-devnull]

Ok so I did some proper testing.

Commands

Identical filtering conditions: specific paths, specific file extensions, a number of exclusions

rg --files -uuu -L [...] | lscolors
fd -LI -t f --color=always [...]

Benchmark (force cold cache)

Conclusion: fd is 10% faster on cold cache

hyperfine --runs 2 --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null' ./hf_rg ./hf_fd
Benchmark 1: ./hf_rg
  Time (mean ± σ):     257.971 s ± 14.641 s    [User: 5.750 s, System: 23.679 s]
  Range (min … max):   247.618 s … 268.324 s    2 runs

Benchmark 2: ./hf_fd
  Time (mean ± σ):     234.401 s ±  0.326 s    [User: 6.046 s, System: 19.817 s]
  Range (min … max):   234.171 s … 234.632 s    2 runs

Summary
  ./hf_fd ran
    1.10 ± 0.06 times faster than ./hf_rg

Benchmark (warmup + warm cache)

Conclusion: rg | lscolors is 7% faster on warm (hot) cache

hyperfine --warmup 1 --runs 3 ./hf_rg ./hf_fd
Benchmark 1: ./hf_rg
  Time (mean ± σ):      4.586 s ±  0.003 s    [User: 3.072 s, System: 7.142 s]
  Range (min … max):    4.583 s …  4.589 s    3 runs

Benchmark 2: ./hf_fd
  Time (mean ± σ):      4.928 s ±  0.223 s    [User: 4.230 s, System: 7.326 s]
  Range (min … max):    4.745 s …  5.176 s    3 runs

Summary
  ./hf_rg ran
    1.07 ± 0.05 times faster than ./hf_fd

Final conclusions

• Linux FS cache is very important and quite short lived/volatile depending on the machine setup; this produced initial manual test results that were not reliable and did not show the objective reality
• cold (frozen) cache: fd is 10% faster than rg
• warm (hot) cache : rg is 7% faster than fd
• for an interactive shell script with the scope to be used by multiple users, a background cache warming is the best option to avoid some potentially long wait times for results: here rg wins, though slightly if we look at raw benched execution times
• rg holds up surprisingly well considering its completely different usage scope