Mobile @ Scale: my experience

Facebook recently hosted its first-ever Mobile @ Scale conference at its hacker headquarters. The conference assembled senior engineers and directors from Twitter, Facebook, Pinterest, Dropbox and LinkedIn into a rather small room to discuss common issues faced by mobile engineering teams. Access to information was impressive and the transparency was refreshing. I was very excited to be a part the crew invited to attend.

Below, I’ll cover what I felt were the high level themes, and some of the more interesting bits, from the jam packed day of talks. I’ll later publish a podcast I recorded which covers the content in more detail!

Silo Problem Solving.
The Facebook VP of Engineering, Cory Ondrejka, was first to point out that we’re all facing and solving similar problems in mobile, it being such a fresh, and fast changing area of tech. The issue is that we’re problem-solving in isolation and often not sharing these stories. The Mobile @ Scale event was meant to start solving this problem and start opening the lines of communication.

Focus on Speed.
No where does speed seem more top of mind than at Facebook. Their release engineering team is building incredible tools for both Android and iOS (many are being open sourced) to ensure builds happen fast, testing is easy, customer feedback is accessible, among other endeavors. Facebook, and also Twitter, are relentlessly focused on further trimming the time between releases and ship continuous value to customers. On Android, Google’s Alpha and Beta programs have allowed for select customers to receive builds on a rapid cadence (daily Alpha builds coming from Facebook). Both FB and Twitter called for Apple to open a similar program for iOS.

Experimentation for all.
I loved the way Facebook and Twitter talked about empowering everyone on the team to invent. They cleared process to ensure that everyone feels a sense of product ownership and are tuned in to thinking about improvements. Twitter talked about how teams are empowered to ship a feature to 1% of users, with no executive level approval needed. Simply ship to 1% and start measuring. Facebook has a similar stance and sounded like they were moving toward a place where each and every developer has this flexibility to build and ship to to a small portion of customers - no gates. Build, ship, measure.

Measurement.
These companies lean heavily on direct customer feedback, crash reports, usability studies, but they lean most on the usage data collected across their products. At LinkedIn, they collect data on 500 unique events in their app and can slice the reports by app version, OS level, device type and more. They compare data weekly to ensure features shipped are moving the product usage in the direction they want. Measurement is crucial in validating new features and the experimental work they push to customers.

HTML5.
Among the mistakes made, an overreach on HTML5 was cited more than once - Facebook and LinkedIn. Both companies have spoken about this before, but it bears repeating. Facebook’s candid take was that the choice to go all-in on HTML5 was a decision that was best for Facebook R&D, but not one that was best for customers. They’ve now refocused on customers, and therefore it’s a mostly native product today, with web views only appearing in less trafficked areas of the app. Similar story for LinkedIn, though they cited that the switch away from HTML5 wasn’t usability but debugging difficulties and high crash rates (due to memory exhaustion). All of the products represented at the conference are native-first applications.

Scaling Teams.
Each of the speakers spoke of a time in which there was only 1 or 2 developers on their mobile team. Some teams, like the Mailbox iOS team, are still small - Mailbox is operating with 2 iOS developers. However, companies like LinkedIn, with 5 products to maintain and 38% of their total traffic on mobile, now have 65 mobile engineers.

For Twitter, the high growth on mobile usage meant that product managers from around the company started turning to the mobile team to drive product innovation. Very quickly, the mobile backlog blew up to a point where it couldn’t be managed by the small team. Their first attempt to correct this, was to loan mobile talent out to other product teams and have them contribute their knowledge to teams which didn’t have mobile experience. That didn’t work. Twitter’s Director of Engineering, Jeremy Gordon, explained that being loaned around to various teams didn’t give the engineers a sense of ownership over any one product. They felt disconnected. What has worked, is building teams where each is responsible for a single feature area and bringing it to all platforms. This is the same model used by Facebook. To increase the mobile talent at Twitter, they trained many of their engineers in mobile, through Twitter University, which provided skills allowing more of their developers to move flexibly between mobile, web and backend.

Scaling Applications.
Another thing that became clear in listening to the engineering team at Facebook, is that the app is big. Their customer scale is massive, but the app has broad scope and they cover a number of platforms. All of this complexity needs to be managed through thoughtful app architecture. Their Android app is broken up in 15 APKs and the iOS app uses Modules to manage feature areas in decomposed chunks. This allows each iOS feature team to build their module independent of other feature teams. Facebook Engineering Manager, Alan Cannistraro, explained that they use URLs to navigate the user between modules (views) and then leverage the same URL scheme in push notifications to route users from a push to the target view in the app. They also use Xcode workspaces to manage all of the projects which make up the application. Alan was formerly at Apple for 12 years.

The guys from Dropbox/Mailbox told a story in which the two companies (formerly separate) came to the same conclusion on how to share client side code across platforms. The shared client code was written in C++ (mostly data layer; sync, cache, etc) and then built into a library which could be used in both their Dropbox Android and iOS app, and was also used as the 3rd party developer sync library. Quite a win for the team.

Motion Design.
Facebook Android Developer, Will Bailey, talked about how he and the team built Facebook Home for Android. He covered an interesting topic; Motion Design. The challenge we face in building mobile apps, is that the design tools we use do not help us convey how motion should be applied in our designs. Will revealed that the design team used Quartz Composer on the Home project to render the designs with full animation applied. The Composer project allowed the designers to tweak animation parameters as they applied finishing polish, which was a great way of communicating the motion design to the developers. To achieve some of the animated spring effects in Home, Will built a spring physics engine, called Rebound, which was open sourced on the day of the conference.

My thoughts.
I think what struck me, is that even my own team has encountered a lot of the same problems, and solved them in similar ways to Twitter, Facebook and the like. Until this conference, I didn’t know that we had stories worth sharing. Now I do, so I hope to continue this trend of sharing that Facebook has graciously kick started. I also found it amazing how honest and open these companies were in sharing their experiences, both proud moments and mistakes. This was valuable. Knowing pitfalls to avoid, and great solutions to adopt, will help my team to build better product. If I was to summarize the day I would say that it reinforced the power of shipping fast, empowering the team to invent and always looking ahead to see how we can solve these mobile scaling issues before they hit.

A truly great day at 15 Hacker Way. Thank you, Facebook.