Ineffective Retry Mechanisms

“Flaky Tests Are Not a Testing Problem. They're a Feedback Loop You Broke Every retry rule in your CI pipeline is a painkiller. It suppresses the symptom, the stock of broken code keeps growing underneath, and nobody feels the pain until the whole system is addicted.<p>I came across this HN post that perfectly illustrates the pattern: https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=46967724<p>Retries, quarantining, adding waits - these aren&#x27;t fixes. They&#x27;re letting CI skip errors temporarily. The real problem: there&#x27;s a growing stock of broken code feeding your pipeline, and no mechanism exists to make the person who introduced it feel the pain.<p>Two reinforcing loops start running:<p><pre><code> RED PIPELINE --&gt; &lt;RETRY&gt; --&gt; GREEN BUILD ^ | | (Long-Term) | | v MORE FLAKINESS &lt;---- HIDDEN BUGS ACCUMULATE </code></pre> R1 &quot;The Addiction&quot;: Every retry makes the light green. But hidden bugs accumulate underneath, making the system flakier, forcing more retries tomorrow. This is textbook &quot;Shifting the Burden&quot; from Donella Meadows.<p>R2 &quot;The Erosion&quot;: Because you don&#x27;t trust CI signal, you lower standards. Because you lower standards, worse code gets merged. Signal becomes even less trustworthy. Repeat until your pipeline is a decoration.<p>The original post asked how QA and engineering should split responsibility. Wrong question. The right question: how do you make the pain of instability felt by the person who introduced it?<p>I HIT THE SAME WALL<p>I built CI to port 973 ROS 2 packages onto two non-officially-supported Linux distros (openEuler + openKylin, RISC-V). Zero upstream support.<p>v1 - Brute-force probe. Pull all 973 packages, let them break. 597 built, 214 dependency gaps and 151 failures mapped. The pipeline wasn&#x27;t meant to pass. It was meant to make every hidden stock visible.<p>v2 - Verification engine. Probe the environment first, identify gaps before building. Stop feeding garbage into the pipeline. Build attempts dropped, success rate went up.<p>v3 - Incremental stock management. Isolate small batches of problems, resolve them one group at a time. Subtraction, not addition.<p>MY SYSTEM IS ADDICTED TOO<p>Here&#x27;s where I punch myself in the face. My own CI has the same pattern. Virtual environments to bypass dependency conflicts. Masquerade rules that spoof package identities. Band-aids.<p>But I know they&#x27;re band-aids. Most teams don&#x27;t. They think retries are solutions. Being aware of the addiction and being consumed by it are two very different things.<p>I can identify the stock poisoning your pipeline. I can design the feedback loop that makes the right person feel the pain. What I can&#x27;t do is force an organization to care. That&#x27;s usually the real bottleneck - not the flaky tests, but the system&#x27;s refusal to let anyone feel the consequences.<p>Repo: https:&#x2F;&#x2F;github.com&#x2F;Sebastianhayashi&#x2F;the_adaptive_verification_engine”