Hidden Issues from Work-Shifting

“Why We’re Sticking with Ruby on Rails at Gitlab A lot of teams use tools that can do this sort of thing OK out of the box. I&#x27;ve set up CI to only run affected tests and use incremental builds in the past. My current project has incremental CI.<p>The core problem is actually dev teams. Every time I&#x27;ve tried to implement incremental CI when I wasn&#x27;t in a position to force it through, people fought against it, complained, moaned etc until higher level management gave in. The problems were:<p>1. People liked the feeling of every test passing every change. It made them feel safe and like they could point the finger at CI if something went wrong, because if every test passed it&#x27;s definitely not their fault, right? But this leads to slow builds and then they complain about that instead. However this is preferable because slow builds are the project manager&#x2F;TL&#x27;s problem to solve, not theirs, so there&#x27;s a built-in bias towards over-testing.<p>2. If there&#x27;s any issue with the build system that causes one test run to affect another in some way (shared state of any kind), then incrementality can cause hard to understand flakes. Sometimes tests will just fail and it&#x27;ll go away if you force a truly clean rebuild. Devs hate this because they&#x27;ve been taught that flaky tests are the worst possible evil.<p>3. Build systems are generally hard-wired for correctness, so if you make a change in a module that everything transitively depends on, they&#x27;ll insist on retesting everything. This is reasonable from a theoretical perspective, but again, leads to very slow builds and people finding ways around them.<p>4. Work-shifting. Splitting stuff out into separate build systems (separate repos is not really the issue here), may appear to solve these issues, but of course in reality just hides them. Now if you change a core module and break things, you just won&#x27;t know about it until much later when that other team updates to the new version of the module. But at that point you&#x27;re working on another ticket and it&#x27;s now their problem to deal with, not yours, so you successfully externalized the work of updating the use sites to someone else.<p>Competent firms like Google have found solutions to these problems by throwing hardware at it, and by ensuring nobody was able to create their own repositories or branches. If your company uses GitHub though, you&#x27;re doomed. No way to enforce a monorepo&#x2F;unified CI discipline in such a setup.”