Frustrating Testing System

“Wrangling 2000 Git Repos at Reddit To those who are swinging towards monorepos, I don&#x27;t think that is a good solution. The reason being is that developers simply cannot be trusted to &quot;do the right thing&quot; on data and module boundaries. Someone comes in new to the project and does something they don&#x27;t know they shouldn&#x27;t. It is the honor system backed by weak linters and tooling.<p>In our monorepo, everyone passes around django orm objects and boundaries are practically non-existent. N+1 queries abound. Tests are full of patching and mocking and are _slow_. Our build takes over an hour to run tests. Someone on team A can and absolutely will mess up what someone on team B is doing. We are now having to spend quarter upon quarter as we define and enforce domain boundaries within the python code base. It is all bolted on checks. Tests are getting worse and people are actively trying to figure out ways around the testing system because it sucks.<p>Compare to my last gig. We had several hundred production repos. Each repo starts from a template with its own build pipeline. All production repos are gated so that any PR must pass tests before it can merge. Any merge has to pass tests before it could be deployed. As the base build processes matured, teams could, at their leisure, pull their services up to the latest and greatest. We even migrated from Jenkins to Buildkite; yeah, it took N pulls into N repos. Not a big deal. Most projects&#x27; tests and builds could get code out to production in under 10 minutes, including all those checks. Due to the network boundary, you couldn&#x27;t accidentally get around someone&#x27;s abstraction. And if one team blew up their build doing something dumb? No problem, it only affects that one team.<p>The argument is &quot;gah, managing all those services!&quot; Keep data behind APIs. Keep APIs backwards compatible. Keep dependencies acyclic. This is _possible_ with monorepos, but you have to do extra work compared to networked services -- yes, when any particular team&#x2F;service can deploy in minutes due to low build system complexity you are winning. Can you get that wrong and make strange cyclic dependencies and introduce performance issues due to network hops? Yeah, of course. However, we were processing, literally, 10s of billions of api requests on this system and teams could work untethered from one another. The new gig does eerily similar software, but is several orders of magnitude slower in their ability to process data and their ability to move new features.<p>yes, yes, you could have networked services and a monorepo and you can leverage tooling like Pants to minimize the testing to only account for changed files. It is just fighting what I have found to be a better model. Keep things separate. Keep things fast to change.”