How to Manage Code Across Many Independent Repositories

Justin Meyer by Justin Meyer

How to Manage Code Across Many Independent Repositories

Justin Meyer Learn the benefits and techniques of splitting codebases across multiple repositories we discovered while building CanJS 3.0.

posted in CanJS ,stable-and-innovative on September 7, 2017 by Justin Meyer

In This Series: Stable and innovative code bases

  1. Stable and Innovative Code Bases
  2. How to Manage Code Across Many Independent Repositories
  3. Removing Side Effects - some juice isn't worth the squeeze
  4. Coping with Stateful Code
  5. How to Integrate Other Libraries using Symbols

The first, and most important step to supporting stability and innovation within CanJS's codebase has been breaking up CanJS into individual repositories, each with its own npm package and semantic version number. In this article, we will discuss:

  • The benefits of independent repositories.
  • How we manage a codebase split across many repositories.

Benefits of Independent Repositories

There are currently over 60 different repositories in CanJS:

Core Infrastructure Ecosystem Legacy
can-component can-attribute-encoder can-connect-cloneable can-ejs
can-compute can-cid can-connect-feathers can-list
can-connect can-construct can-connect-ndjson can-map
can-define can-control can-connect-signalr can-map-backup
can-route can-deparam can-construct-super can-map-define
can-route-pushstate can-dom-events can-define-stream can-validate-legacy
can-set can-event can-define-stream-kefir can-view-href
can-stache can-namespace can-define-stream-validatejs  
can-stache-bindings can-observation can-element  
  can-param can-fixture  
  can-reflect can-fixture-socket  
  can-simple-map can-jquery  
  can-symbol can-kefir  
  can-types can-ndjson-stream  
  can-util can-observe  
  can-validate-interface can-react-component  
  can-view-callbacks can-reflect-promise  
  can-view-live can-stache-converters  
  can-view-model can-stream-kefir  
  can-view-nodelist can-validate  
  can-view-parser can-validate-validatejs  
  can-view-scope can-vdom  
  can-view-target can-view-autorender  
    can-view-import  
    can-zone  
    react-view-model  
    steal-stache  

 

Organizing CanJS into individual repositories and packages has many benefits.

The obvious advantage is that pieces can be used without the whole. You can choose to use CanJS’s observables or can-fixture without the rest of the framework. You could even mix and match CanJS libraries with other libraries like React quite easily.

However, the main benefit is that independent repositories improve CanJS’s stability — one half of CanJS’s mission. This is because independent repositories make it easier to upgrade more frequently. For example, let's compare:

  • Upgrading a 2.3 app, which was not organized in individual repositories, to
  • Upgrading a 3.0 app.

Despite making relatively few breaking changes, and providing a migration guide, upgrading from CanJS 2.3 to 3.0 looks like a big step:

 

 

But if you break that step down, CanJS 2.3 is mostly CanJS 3.0 with a bunch of bug fixes, a heap of new features, and a few breaking changes. Most of the difficulties upgrading are the breaking changes, which account for the majority of the upgrade step size:

 

 

To get all of those bug fixes and new features in 3.0, you have to take on those breaking changes from 2.3 all at once. Depending on your company culture, and scale of your application, this might not be easy.

Going forward in CanJS 3.0, packages are released independently of each other. You can upgrade to bug fixes and new features immediately and delay breaking changes (example: can-route 4.0.0) until later. You can upgrade breaking changes in steps too. For example, you might upgrade to can-route 4.0.0 one month and can-component 4.0.0 the following month. CanJS 3.0’s upgrade path looks like:

 

 

Independent repositories also mean that legacy libraries, like can-ejs can continue living through community-driven fixes and releases. They don’t die simply because they are no longer included in the core CanJS build.

In short, independent repositories and packages:

  • Allow users to get bug fixes and features without forcing them to accept breaking changes.
  • Support asymmetrical development, allowing the community to focus on what's important and experiment easily.
  • Helped us make over 150 releases since CanJS 3.0.

Managing Independent Repositories

Managing so many repositories would be a difficult task without great tooling. To make this easy, we use:

  • DoneJS's plugin generator to add automated testing, builds, and publishing.
  • The canjs/canjs repository as an integration test.
  • GreenKeeper to let us know if we break any upstream packages.
  • Landscaper to make changes across multiple repositories at once.
  • ZenHub to manage issues and create epics across multiple repositories.

DoneJS's plugin generator

DoneJS's plugin generator makes it easy to author a JavaScript open source project. It creates the files and scripts necessary for:

  • Automated tests
  • Continuous integration with TravisCI
  • <script>, AMD, and CommonJS builds.
  • A publish script that runs the tests, performs the build, checks in the dist in the github tag, and publishes to npm.

Walk through the DoneJS plugin generator guide to learn how to create your own plugins.

Integration tests with the CanJS repository

While CanJS is broken out into individual repositories and packages, there’s still a need to test for problems when combining packages. The canjs/canjs repository is used to load every package’s tests and run them all at once within each supported browser. We also have additional integration tests to make sure our guides and production builds work.  

The canjs/canjs repository is also used to establish specific versions of every package that are verified to work together. Organizations can upgrade to a specific version of CanJS by using the same dependencies. The latest version of CanJS documents its package versions here

Test breaking upstream packages with GreenKeeper

GreenKeeper.png

We use GreenKeeper.io to know if we’ve broken any upstream dependencies. When we make a new release, GreenKeeper makes pull requests to our repositories using that release, which runs the repositories tests. We get an email when those builds fail.

Make changes across multiple repositories at once with Landscaper

landscaper.png

Landscaper is a command-line tool for making sweeping changes to any number of projects using code mods. If we want to change the license copywrite year across all 60 repositories, we write a code mod and use landscaper to submit a pull request to all 60 repositories.

landscaper https://gist.github.com/andrejewski/8d0b4927f73978e78b0105f84ad8abd4

Manage issues across repositories with ZenHub

zenhub.jpg

We use ZenHub to manage our issues across multiple repositories, including adding complexity scoring and combining issues into epics. The following shows all the issues assigned to the CanJS 3.Next release:

CanJS_3_NEXT_Epic_·_Issue__3277_·_canjs_canjs.png

Conclusions

Breaking up CanJS into many repositories has been a huge effort. Even with the tools above, the simplicity of a single repository can sometimes still feel appealing. But the results have so far been overwhelmingly positive.  We've been able to add three to four times the number of features and bug fixes in the last 10 months than the previous 10 months.  Multiple repositories also forced us to write code that is more thoughtfully constructed and better architected. We will see how in the next articles in this series:

Create better web applications. We’ll help. Let’s work together.