News about Lychee and why recursive link checking is not trivial.

Challanges:

One is named "distributed termination detection": know when you're done. "The classic solutions (Dijkstra–Scholten, token passing) just don’t map well onto Tokio’s channel-based world.

Another is the cycle because it's a DAG.

Then there is backpressure with the checker and the sender to the channel. If that channel is full, the response handler blocks; if it blocks, no responses are consumed; if no responses are consumed, no request slots free up.

Deduplication Races can occurs because the links are checks aynchrono

Leaky abstraction because the recursion spreads everywhere: reponses need to carry discovered links, requests need a depth, the collector need to understand recursive inputs, stats and formatters need to handle duplicates.

There is hope though because there is progress!

None of those are lychee problems. They’re hard concurrent-systems problems. We just lacked the vocabulary to talk about them, and while I wasn’t looking, those primitives got built. Sometimes the most important code you write for a feature is the code that never mentions the feature at all. So no, I don’t think we failed. We made progress by stumbling into the right direction.