Monorepo: 11X Faster Development Turnaround

Having the ability to make a code change and quickly see the result is crucial. Our team often experienced the frustration of waiting extended periods between making code changes in our shared codebase and seeing those changes reflected in the UI. To address this issue, we implemented several enhancements, including the consolidation of multiple git repositories into a single monorepo. This change resulted in a remarkable 11X speed boost for iOS and an astounding 86X improvement for Android. In this post, I'll walk you through how we conceptualized and executed this idea.

Background: Introducing Kotlin Multiplatform (KMP)

Back in 2019, our small product team faced challenges supporting both iOS and Android for our mobile apps. For instance, we maintained separate date calculation logic on both iOS and Android, leading to tricky issues on one platform while the other is working as expected. That's when we adopted Kotlin Multiplatform (KMP). KMP allowed us to write less code, develop features faster, and significantly reduce bugs.

Around the same time, SwiftUI and Jetpack Compose were emerging. We saw an opportunity to transition to a new architecture using Declarative UI + Kotlin Multiplatform. You can find more details in this article, but in essence, we use Kotlin Multiplatform for shared logic (e.g., models and network code) and SwiftUI & Jetpack Compose for user-facing interfaces. This approach ensures feature parity between iOS and Android without sacrificing user experience, and it has proven highly effective for our team's structure.

Since we initially introduced KMP as an experiment, we created a separate Git repository and configured our two mobile repositories to reference it. Here's how we structured the repositories. The LC iOS and LC Android projects were using a specific version of LC Common as a library.

The Problem: Bottlenecks and Inefficiencies

The setup worked well initially, but as our codebase grew and required frequent changes, we encountered several challenges.

A. Developer Turnaround Overhead: Making changes that affected both the common code and client-specific code required multiple steps as below:

1. LC Common – Build on the CI (8 minutes)
2. LC Common – Code Review (<24 hours)
3. LC iOS or LC Android – Update & sync LC Common (10-20 seconds)
4. LC iOS or LC Android – Build & run (30-37 seconds)

In this example, you end up waiting for up to a day and 9 minutes to see LC Common changes reflected in the UI layer. If we exclude code review step, since it can vary from time to time, you still have to wait almost 9 minutes. As you know, when developers have to pause their work and wait, it's not just that specific amount of time they lose; it often leads to even more productivity loss as their minds start to wander.

B. Compatibility and Bug Challenges: It was relatively easy to introduce critical issues when working on LC Common. For instance, imagine you made a change for LC Common and LC iOS code as an iOS engineer. You had to remember that there is an Android counterpart that requires testing as well. As a result, we often experienced issues on the other platform. Our CI process was limited to individual repositories, lacking the automated tests for changes across repositories.

C. Redundant MRs: Any cross-platform changes, regardless of their size, forced us to create at least three merge requests (MRs). To illustrate, consider a scenario where you simply updated text within LC Common. This would require one MR to modify the text within LC Common and an additional two MRs to integrate the updated LC Common into both the LC iOS and LC Android repositories.

D. Release Coordination Issues: We made the decision to commit the compiled binary to the repository to support Swift Package Manager with the minimum amount of effort. This approach worked well when the binary was small and we had just two engineers. However, as multiple years passed and our team expanded, we found ourselves dealing with merge conflicts almost every time we released new builds.

E. Sluggish Developer Onboarding: Due to the issues mentioned above, the size of our files ballooned over time, reaching an astonishing 13 GB for our Git repository. This meant that new engineers joining our team faced hours of wait time just to clone the repository from scratch. 🤦‍♂️️

F. Supporting Apple Silicon: This wasn't necessarily an issue exclusive to separate repositories. However, when Apple introduced Apple Silicon, we found ourselves in need of a third target – iOSArm64Simulator – to run tests using simulators. Given the already substantial size of our repository, adding an additional target was something we couldn't afford, as it would further inflate the repository's size.

The Solution: Merging into a Monorepo

To address these challenges, we made the decision to migrate individual repositories (LC iOS, LC Android, LC Common) into a single monorepo, and we called it LC App. We drew inspiration from the Mobile Native Foundation's Discussions and recognized the advantages it offered.

How We Executed the Plan

Since the plan involves merging three repositories into one, we might think it's as straightforward as merging three text files into one larger text file, right? Well, as it turns out, it's a bit more complex than that.

Phase 1: Proof of Concept

Before committing the team's effort to the project, we wanted to ensure its value. Our approach involved creating a quick and dirty sample project that could provide measurements to inform our decisions.

This phase also served as a valuable learning experience to understand the necessary future steps. Here's a breakdown of the steps we took to validate the project:

1. Create a new Kotlin Multiplatform project using the latest official template. This allowed us to leverage the most recent and recommended mechanisms for linking binaries.
2. Copy files from LC Common into a subfolder named common.
3. Link LC Common to the sample iOS project. During this step, we measured the build time and confirmed its efficiency. We also verified that the compile cache functioned as expected, with a focus on improving partial compiling time.
4. Copy files from LC Android into a subfolder named android.
5. Link LC Common to LC Android. We began with Android because both platforms use Kotlin. Once again, we measured the build time and ensured its speed.
6. Copy files from LC iOS into a subfolder named ios.
7. Link LC Common to LC iOS. During this process, we updated the Kotlin Multiplatform linking mechanism from packForXcode to embedAndSignAppleFrameworkForXcode. Additionally, we had to address a few third-party libraries that were incompatible with the new mechanism. We were fine with just removing them for the purpose of this phase. We could revisit this during the actual execution to migrate those dependencies.

Phase 2: Team Discussion

After our Proof of Concept demonstrated the project's potential, our team's leaders and engineers came together to discuss the way forward and share their feedback and concerns.

One concern that surfaced was the increased code exposure for engineers. For instance, if you were previously an iOS engineer, your focus was primarily on LC iOS and LC Common, with no need to worry about LC Android. However, in a monorepo setup, you gain access to all mobile code. This might make it somewhat difficult to locate specific files. Additionally, it could potentially raise security considerations.

As we explored this topic further, we reached a consensus that the additional exposure wasn't overly concerning.

Phase 3: Engineering Planning

Before diving into the process, we took the time to consider several important factors and set up an engineering plan. Here's what we had in mind:

• Continuous Development during Transition: We didn't want to disrupt our engineers' ongoing feature development. So, we leveraged Git submodules as we set up the new repository. We also planned to migrate these submodules into the parent repository.
• Preserving History: We were determined to keep our commit history. Especially in emergency situations when you're trying to understand the state of the codebase, having access to the commit history and extra context is invaluable.
• Size Reduction: Given our already substantial repositories, merging them into one could potentially triple the size. Our goal was to keep the repository compact.

Phase 4: Execution

Here are the steps we took to make it happen:

(1) Set Up a New Repository

We created a fresh repository using the Kotlin Multiplatform template. Then, we added three sub-modules as below. Again, the goal was to continue feature development as a team even during the migration except when it was absolutely necessary.

├─ ios     // LC iOS
├─ android // LC Android
└─ common  // LC Common

(2) Utilize the Local LC Common for Building

Then, we got to work to make LC Common build successfully with both iOS and Android projects. Fortunately, our previous experience during the proof of concept phase made this step relatively straightforward.

(3) Update the CI

We wanted to stick with our existing CI to ensure our changes wouldn't bring any surprises. Therefore, we revisited the current jobs and made necessary tweaks.

• Lint
  • Ktlint (Kotlin): We were running Ktlint separately for LC Android and LC Common. However, it made more sense to run it once against the root folder. During this process, we also discovered that these two projects were using slightly different Ktlint rules, so we made changes to make it consistent.
  • Swiftlint (Swift): We were running Swiftlint against LC iOS. We made an adjustment to run it against the root folder. While this change didn't provide immediate benefits, it aligned with our mental model of running linters against the root folder. This adjustment is expected to be helpful for future engineers, including myself.
• Test
  • LC Common unit tests: No adjustments were necessary.
  • LC iOS & LC Android tests: Although we didn't modify the CI job itself, we were able to use the updated LC Common when running these tests, as it became part of the same repository. This enabled us to identify bugs that couldn't be detected by LC Common's unit tests.
• Deploy
  • LC iOS: Archive and upload to TestFlight.
  • LC Android: Build and upload to Google Play Store.

(4) Communication

This was the final stage of the initial migration. We needed to remove the links between the main repository and the sub-modules and archive the individual repositories. Following this change, engineers had to adjust their local setups to align with the new repository. Recognizing that this transition might introduce some disruption to development, however minor, we carefully scheduled the migration and allowed for a buffer to address any unexpected issues.

Here's one of the announcements we posted in the team chat:

Monorepo Progress Update

Hey team, as we have been testing Monorepo recently, here' the update:

- Monorepo now has all CI phases, including linting, building, testing, and deployment.
- We're covering more tests in less total time, and the bonus is we no longer need to use Rosetta mode for Xcode 14.
- Mark your calendars for an important date: December 19th, Monday. On this day, we'll archive the existing separate repositories and migrate their develop branches into the Monorepo. Since we're removing the submodule bridges, any unmerged branches after this date will require you to cherry-pick your work into the Monorepo.
- If you haven't already, now is a perfect time to clone the Monorepo and start experimenting with it. We will talk more about this during the upcomoing Engineering Team meeting (tomorrow). Please let me know if you have any questions or concerns.

(5) Clean Up

We removed the links between the main repository and the sub-modules.

As we wanted to keep the commit history, I found this helpful article. By following the script it offered, we successfully retained our entire history, including the initial "first commit" dating back to 2016.

We also renamed git tags to make it easy to find specific previous releases. Previously, LC iOS and LC Android used distinct tag names like 4.1.12. However, the integration into a single repository led to conflicts. We added platform prefixes to the version numbers. For example, LC iOS 4.1.12 became ios/4.1.12. For your reference, here's the script we ran to make it happen.

# After importing LC Android tags
git tag -l | while read t; do n="android/${t}"; git tag $n $t; git tag -d $t; done
# After importing LC iOS tags
git tag -l | while read t; do if [[ $t = android* ]]; then continue; fi; n="ios/${t}"; git tag $n $t; git tag -d $t; done

We also used BFG Repo-Cleaner to remove old binaries. We identified files surpassing 10 MB in size, and confirmed that the majority of these files were indeed Kotlin Multiplatform compiled binaries. There were only 5 files that did not fit this category, and we addressed them separately. Here's the script we used to remove the compiled binaries, resulting in a substantial savings of 11 GB.

bfg --strip-blobs-bigger-than 10M my-repo.git

Conclusion

Following these adjustments, we believe the migration was an overall success. Prior to the transition, it took nearly 9 minutes for any shared code changes to be reflected in the UI. Now, it takes only 47 seconds (11X faster) on iOS, and 6 seconds (86X faster) on Android.

Here's some feedback from a team member:

It is such a seamless experience to now be able to quickly edit LC Common and Android code at the same time and test it instantly. Awesome job at #CreatingMargin & #SimplifyingProcesses!

As we look ahead, our team iterated on the migration and further improve our processes. Here are the areas are focusing on:

1. Implementing selective test runs based on the code changes made. For instance, if you've only modified iOS UI code, there's no need to run Android UI tests.
2. Developing the capability to filter commits by platform. In our previous setup with separate repositories, the commit history exclusively displayed changes for your specific platform. With our new monorepo, you can see commits from iOS, Android, and Common. We're aiming to find an easy way to filter platform-specific changes efficiently.