following the post "5 years trying to add recursion to lychee" https://shaarli.lyokolux.space/shaare/STTHtQ

The key takeaway is: they didn’t find a clever trick we missed. They were built as crawlers from the very first commit, and I initially built lychee as a stream.

Why it's still hard and not solved

• the other projects started as crawlers; lychee started as a stream
• the frontier and the rate-limiter must be different objects
• single-threaded runtimes get dedup for free.

it’s a harder problem than just “copy what they do,” because most link checkers didn’t start with uncompromising performance as their top goal.

So the other projects "made it a part of the architecture from the beginning, and they leaned on a runtime (providing conveniences like a WaitGroup, a joinable queue, an idle promise) that solved termination without solving “distributed termination detection.”