Betting on the web: Ensuring fine-tuned performance for dynamic web apps (Adobe)

In this talk, Charlton Roberts of Frame.io, now part of Adobe, discusses the challenges of achieving high-performance web app interfaces and the inadequacy of the traditional "make it work, make it right, make it fast" framework for their dynamic video collaboration tool. Roberts emphasizes the need for "fluid UI," which prioritizes desktop-level performance, characterized by being instant, smooth, and coordinated. He shares tactics such as using local state for immediate response, leveraging CSS for animations, and ensuring coordinated actions to maintain the illusion of seamless interaction. Frame.io's approach includes setting performance as a feature, requiring any new updates to meet fluid UI standards before merging, and dedicating resources to anticipate and solve performance issues. Roberts concludes by highlighting the importance of shared vision, expectations, and dedicated resources in achieving and maintaining a high-performance web application.

• The framework for building software "make it work, make it right, make it fast" was popularized by Kent Beck.
• It is commonly used in test-driven development.
• The typical development process involves spending most of the time on "make it work," then "make it right," and finally "make it fast" before a deadline.

"How many of you, show of hands, have seen this framework for building software: make it work, make it right, make it fast?"

• This quote introduces the framework and asks the audience for their familiarity with it.

"It was popularized by Kent Beck and some of his writings about test-driven development."

• This quote credits Kent Beck for popularizing the framework and connects it to test-driven development.

• Frame.io found that the "make it work, make it right, make it fast" framework was not effective for their high-performance web application.
• The speaker, Charlton Roberts, has extensive experience building high-performance websites and web apps.
• Frame.io, acquired by Adobe, is working on version 4, which aims for desktop app-level performance on the web.

"Yeah, when building a really high-performance, very dynamic web app in Frame.io, we found out that this framework just didn't work for us."

• This quote explains that the traditional framework did not suit the needs of Frame.io's web application development.

• Frame.io is a video collaboration and asset management tool.
• The new version is built on Next.js and Vercel.
• Frame.io's goal was to achieve desktop-level performance for web interfaces.
• Customers, such as Hollywood filmmakers and brands like Ford, require high-quality content with precise frame rates.

"With version 4, in addition to a bunch of the features that we had planned for this next generation, everything that our customers were asking for us around metadata and asset management, one of the really aspirational principles for us was we wanted desktop-level performance, desktop app-level performance on the web."

• This quote emphasizes the importance of high performance as a key principle for Frame.io's version 4.

• 60 frames per second is considered the standard frame rate for the web.
• One frame at this rate lasts 16 milliseconds.
• Frame.io focuses on what can be achieved within each 16-millisecond frame to ensure smooth performance.

"These are two numbers that I think every web developer should know by heart: 60 frames per second and 16 milliseconds."

• This quote highlights the critical performance benchmarks for web developers according to Frame.io's standards.

• The traditional development process did not result in a satisfactory product for Frame.io.
• Frame.io experienced delays and "jank" in their product using the old framework.
• The company decided to prioritize the feel of the product as much as its functionality throughout the development process.
• Performance is considered a feature at Frame.io, and if a product doesn't feel good, it's not considered to be working.

"What if how it feels is as important as how it works the entire way through the process? And we started to reimagine a development flow and a bunch of principles and how we think about shipping software around this idea that how it feels is as important as how it works."

• This quote conveys the shift in Frame.io's development philosophy, where the user experience and performance are as important as functionality.

• In 2018, Apple's WWDC conference talk "Designing Fluid Interfaces" inspired Frame.io's founder, Emory Wells.
• The talk led to a new vision for Frame.io's product development, emphasizing the importance of fluidity and performance.

"In 2018, Apple, at a WWDC conference, gave a talk called 'Designing Fluid Interfaces.' If you've never seen it, I highly encourage you to watch it."

• This quote suggests that Apple's talk had a significant influence on Frame.io's approach to designing user interfaces and highlights its importance to their development team.

• Discusses the importance of creating a fluid user interface that feels like an extension of the user's mind.
• Emphasizes the effort required to maintain the illusion of simplicity in software use.
• The speaker introduces the concept of "never drop the illusion" which highlights the seamless experience for the user and the complex work behind the scenes.

"It feels like you're superhuman, right? I think hopefully you've been there where you just feel like it's moving as fast as you can think. We want to keep up that illusion the entire time the user is using our application."

• This quote captures the essence of the desired user experience: software that keeps pace with the user's thoughts, maintaining the illusion of simplicity and speed.

• Introduces the term "fluid UI" and its significance in building high-performance web applications.
• Outlines three core principles: instant, smooth, and coordinated.
• Argues that mastering these three principles results in an application that feels good to use.

"There are a lot of principles, but there are three of them that we probably spend 80% of our time on, and I would argue if you get these three right, you're going to have an application that feels good."

• The speaker emphasizes the importance of focusing on the three main principles, suggesting that they are critical to the success of an application's user interface.

• Describes the "instant" principle as the application's ability to respond to user actions within 100 milliseconds.
• Explains that this response time creates the perception of immediacy for the user.
• Mentions Interaction to Next Paint (INP) as a performance metric related to this principle.

"You have 100 milliseconds to respond to the user and let them know that you know what they're trying to do. At 60 frames per second, that's six frames. That's your threshold, that's your limit."

• This quote outlines the timeframe within which an application must acknowledge user input to maintain the illusion of instantaneity.

• Details the "smooth" principle, requiring animations to run at 60 frames per second without dropping or skipping frames.
• Suggests that smooth animations contribute significantly to the perception of a fluid user interface.

"The second is smooth. This one's a little bit more self-evident. It's the fact that animations have to run at 60 frames per second."

• The quote explains the standard for animation smoothness, which is crucial for a seamless user experience.

• Defines the "coordinated" principle as the harmonious and unified response of the UI to user inputs.
• Uses a metaphor of energy and motion to describe how user actions should trigger coordinated UI responses.

"Coordinated is the idea that everything happens in one sort of same fluid motion. The metaphor I use a lot with the team is when the user does a click event or a keyboard event, something like that, they are putting energy into the system, into our UI, and the UI needs to respond with one motion."

• This quote describes the concept of UI elements moving together in response to user interactions, creating a sense of unity and fluidity.

• Discusses tactics to ensure the UI responds instantly, such as using local state and optimistic updates.
• Highlights the use of app router and router cache to achieve fast navigations.
• Mentions React compiler as a potential solution for optimizing performance and reducing unnecessary re-renders.

"One of them is making sure you don't be scared of local state and use state and use optimistic local to your component for updating instantaneously if you can."

• The quote recommends using local state management to achieve instant feedback in the UI, which is a key part of maintaining the illusion of immediacy.

• Provides a real-time example of the instant principle by demonstrating a user interaction with the Frame.io interface.
• Describes the process of updating the UI within the six-frame threshold during a folder activation.
• Explains how the team uses frame-by-frame analysis to debug and ensure the interaction is instant.

"And then everything else, breadcrumbs, the thumbnails in the bottom right, everything else is going to happen also within about five or six frames."

• This quote illustrates the application of the instant principle, where all elements of the UI respond to a user action within the specified timeframe for perceived instantaneity.

• Offers an explanation of frames per second using an animation example.
• Emphasizes the importance of intermediate states in animations to maintain smoothness.

"It's how many times we see the box in the intermediate states. That's what we're going to look at."

• The quote clarifies that smooth animations are judged by the number of intermediate states visible to the user, which ensures a fluid visual transition.

• Discusses the concept of smooth animation in an application's user interface.
• Explains the process of a box moving across the screen in an animation.
• Describes the issue of dropped frames and how it affects the perception of smoothness.
• Offers solutions to prevent dropped frames, such as using useTransition in React.

"So we're starting off frame one, frame two, frame three, frame four.. Oh, now we're asking the browser to do something really expensive, and it's got to go figure out, fetch some data, something like that.. Well, time moves on, so now we're in frame five, but the box is still in position four.. Now we get to frame six, the browser's freed up, it can catch up.. Okay, let's jump to frame six.. We've dropped frame five here, right?."

• This quote describes a scenario where a frame is dropped during a browser animation because the browser is busy with an expensive task.

"Use transition is just a way to tell React, 'I have something really important and something really expensive, and I want you to do the important thing first and the expensive thing later.'"

• Explains the use of useTransition in React to prioritize tasks and improve animation performance.

• Emphasizes the preference for CSS solutions over JavaScript for smoother animations.
• Highlights the importance of not unmounting and remounting elements unnecessarily.
• Mentions Next.js tools like parallel routes and layouts to help with this issue.
• Describes the benefits of CSS, including smaller bundle sizes and better performance.

"If there's one takeaway I could give you from migrating a 400,000-line codebase from the pages router to the app router, it would be to prefer CSS solutions everywhere if you can."

• This quote advises prioritizing CSS solutions for performance improvements when refactoring large codebases.

"But this one can be really tough to spot because often it's so fast you don't even see it, but if you slow it down in these slower screen recordings, you can actually see a flicker, and it'll let you know that something's unmounting and remounting."

• Highlights the difficulty of detecting unnecessary remounts without the aid of slow-motion screen recordings.

• Discusses the concept of coordinated animations for a seamless user experience.
• Describes the process of identifying and fixing a bug in an animation sequence.
• Explains the importance of timing in animations, where multiple changes should occur simultaneously.
• Points out a specific bug where an avatar disappears for a few frames during a transition.

"The first thing is we're going to show that active state of casting auditions very quickly.. We're going to blur the search projects on mouse down.. I think we actually activate casting and auditions on mouse up, but we should probably move that to mouse down."

• Describes how user interactions like mouse down/up events should be coordinated with visual feedback for a smooth experience.

"That's very important.. That's what we mean by coordinated.. If they happened at different times, it would feel janky."

• Emphasizes that the coordination of animations is crucial to avoid a disjointed and janky user experience.

"So there's a bug here.. I certainly didn't see it until you step through it.. And this is the type of thing you can catch when you're looking for coordinated actions."

• Illustrates how stepping through animations can help identify bugs that disrupt coordinated actions.

• Discusses the challenge of integrating fluid animations into the broader application.
• Describes the struggle of having features interfere with each other, leading to a janky overall experience.
• Talks about the importance of a cohesive approach to feature development.
• Introduces the concept of 'instant infinity' and the importance of optimizing the right work.

"Features were interfering with one another, and we'd get this death by a thousand cuts where this team would ship something, this team would ship something, and then you look back and all of a sudden, everything's janky again."

• This quote highlights the problem of a lack of coordination between teams leading to a decline in overall application performance.

"Instant is a little bit different.. It assumes you're hitting that six-frame threshold, but it's more of a question: how are you going to keep it instant"

• Discusses the goal of maintaining 'instant' performance, suggesting that achieving a certain frame rate is not enough; ongoing performance optimization is necessary.

• Local changes must be prioritized over other expensive operations.
• Planning for scalability is crucial, anticipating that customers might try to make lists on the page infinitely long.
• Strategies range from setting limits to dedicating resources to optimize specific features like the "finder" for handling millions of assets.

"With infinity, it's the idea that everything on the page that's a list, a customer someday is going to try to make it infinite."

This quote highlights the importance of planning for scalability in design, acknowledging that users may push the system's limits.

"We have some of our most talented engineers, some of them are here today, that spend all day tuning that finder to make it fluid UI compatible and also handle millions of assets."

The quote emphasizes the dedication of resources to ensure that key features can handle extensive use without compromising performance.

• Addressing fluid UI issues by managing concurrent operations since JavaScript is single-threaded.
• Three strategies: deferring tasks, anticipating actions, and offloading tasks.
  • Defer: Delay non-essential operations, like loading data for a context menu.
  • Anticipate: Move operations earlier using tools like prefetch in Next.js.
  • Offload: Move tasks off the main thread, such as uploads to a web worker.

"JavaScript is single-threaded, that can't happen. So there's three ways to account for that: you can defer, make one happen later, can it wait?"

This quote explains the need for task management strategies in a single-threaded environment to prevent UI issues.

"All of our uploads we move to a web worker, just get it off the main thread, don't let it compete for resources."

The quote stresses the importance of offloading tasks to maintain fluid UI by not overloading the main thread with resource-intensive processes.

• Fluid UI regressions in production are a pull request (PR) blocker to prevent performance degradation.
• Feature flags are used to ensure new features are fluid UI compliant before full integration.
• Regular audits of the app are conducted to identify and address recent changes that might affect performance.
• Teaching the team to perceive performance in milliseconds and frames is crucial for maintaining fluid UI.

"You cannot merge your PR if you're introducing something that makes something less instant, less smooth, less coordinated."

This quote underscores the policy that PRs causing any regression in UI fluidity are not allowed, ensuring consistent performance standards.

"The feature flag can only come off when that feature, when turned off or turned on, is fully fluid UI with the rest of the application."

The quote details the use of feature flags as a control mechanism to ensure new features meet established fluid UI standards before they are fully released.

• Fluid UI considerations take up about half the development time.
• Equating the feel of the app with its functionality in terms of time and resources allocated.
• Utilizing the right tools, such as stepping through screen recordings and using Storybook for isolated component performance.
• Identifying team members or teams equipped to handle performance issues.

"We now account for fluid UI as being about half the time we spend on stuff."

This quote indicates that significant development time is dedicated to ensuring the fluidity of the user interface.

"If performance is a feature, you need someone that delivers it as a feature, that monitors it and tracks it and works with teams to help them deliver it for their features."

The quote highlights the need for dedicated roles or teams responsible for overseeing and ensuring application performance.

• A specialized team, such as Frame.io's Interface Architecture team, focuses on app-wide performance.
• This team assists other teams in meeting deadlines without compromising fluid UI principles.
• They also look for overarching improvements that can enhance the performance across the entire app.

"Now we have this team to go embed with them and help them hit the deadline but also help them meet these principles."

The quote explains the role of a specialized team in supporting other teams to adhere to fluid UI principles while meeting project deadlines.

"It's a team that looks for step function improvements across the whole app."

This quote describes the proactive approach of seeking significant performance enhancements that benefit the entire application.

• The end goal of a fluid UI is to be instant, smooth, and coordinated.
• Shared expectations include not allowing PRs with regressions and planning for development time.
• Dedicated resources and clear plans are in place for when teams encounter obstacles in maintaining fluid UI standards.
• Continuous improvement is emphasized, with pride taken in the development of these principles.

"We now have this shared vision that the end goal is clear: we want it instant, smooth, and coordinated."

This quote conveys the ultimate objective of achieving a fluid UI that delivers an optimal user experience.

"These principles have worked really well for us. They help us really get close to never dropping the illusion, and our idea of desktop performance on the web."

The quote reflects on the success of the established principles in approaching the ideal of seamless desktop-like performance in a web application.