performance

[fdncred]

I've been working on ways to make globbing with wax a little faster with rayon and I keep coming up empty. I'm beginning to wonder if it needs to be built in somewhere so that it globs a directory and then all the directories inside get their own thread to glob their contents, up to some limit. Any thoughts on this?

[olson-sean-k]

make globbing with wax a little faster with rayon

Yeah, I've also thought about this a bit. Wax depends entirely on walkdir for this, so I think any such features would need to be implemented upstream. The major exception to this is filtering, as Wax is basically a glorified filter atop walkdir. #49 is a great example of a serious performance problem concerning this filtering, wherein directory trees were not discarded and would be read from the file system for no reason (despite what the documentation claims). I just landed a fix for that in 76afaa1.

[fdncred]

I'm wondering how hard it would be to switch from walkdir to jwalk which already supports streaming, sorting, threads, etc. and is also built on walkdir and ignore.
https://github.com/Byron/jwalk

[olson-sean-k]

Oh, cool, I wasn't aware of jwalk! Thanks for pointing it out! It's not clear to me how easy it would be to replace walkdir with jwalk, but I'll take a look.

[milesj]

Just chiming in that we're seeing slow performance for globs like apps/*/*. Would this be related to walkdir?

Context: moonrepo/moon#1721

[milesj]

I tried switching to jwalk (milesj#1) but the performance was negligible.

I think to actually take advantage of jwalk's parallelism and wax's filtering, we need to implement this: https://docs.rs/jwalk/latest/jwalk/type.WalkDir.html#method.process_read_dir

But the way the walker is setup in wax is a bit too confusing for me at the moment :P

[olson-sean-k]

Sorry for losing track of this and thanks for all the efforts to fix it!

...we need to implement this: https://docs.rs/jwalk/latest/jwalk/type.WalkDir.html#method.process_read_dir

Yeah, that API definitely seems important given the way filtering works in wax.

I tried switching to jwalk (milesj#1) but the performance was negligible.

Interesting. You previously mentioned issues with the pattern apps/*/*. Did you perhaps mean apps/**/*? I ask because I would expect the former to stop the walk fairly quickly.

My hunch is that the walk itself is not the bottleneck here. At the moment, wax matches all paths against the pattern while walking even if the pattern is exhaustive (meaning that once a path matches, then all subpaths must also match). It is absolutely possible to improve this problem, but it relies on accurate determination of exhaustive patterns.

I'm actually in the middle of a major refactor and rewrite. Among many other things, I believe it will make determining exhaustiveness a bit easier and more reliable. For exhaustive matches, the walk could be cancelled and delegated to a walk that does not compare against any patterns at all.