Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Graphics Pipeline

The graphics pipeline consists of three phases: the layout phase that turns the Widget Tree into the Widget Layout Tree, the redenr phase that tuns the Widget Layout Tree into a Display List, and the draw phase that turns the Display List into screen pixels.
Figure 1: Graphics pipeline

The SmolPhone uses a commonly-used graphics pipeline to turn a widget tree into pixels on the screen. The pipeline consists of three main phases depicted in the Figure 1:

The layout phase computes the size and the position of each widget on the screen. A size and a position form a bounding box. To lay out the widgets, the tree is traversed twice: first in a bottom-up fashion and then in a top-down fashion. The first traversal computes the minimum size of each widget based on the minimum sizes of its children. The second traversal distributes the remaining space of the widgets to their children and computes the positions of the widgets relatively to their parents. Flutter and SwiftUI use dedicated widgets to express layout constraints. Conversely, we attach layout constraints to each widget. This avoids adding lots of widgets just for layout purposes. This also has the advantage that layout constraints can be removed if the bounding boxes can be determined during compilation.

The render phase generates a list of graphics primitives, commonly named the display list, from the widgets. Graphics primitives are rectangles, images, and glyphs. This phase is responsible for converting texts into glyphs and uses a theme to render the widgets. Modern frameworks enable widgets to cover multiple layers simultaneously, allowing them to partially overlap their children. Also, a widget can sometimes exceed the bounding box of its parent. We chose a tiling paradigm in which every widget is contained within the bounding box of its parent, doesn’t overlap its siblings, and is always rendered above its parent. It is claimed that this paradigm is simpler to implement by the Oberon project. Generating the display list requires only a preorder traversal of the widget tree.

The draw phase processes the display list and draws its graphics primitives on the screen. Modern graphics pipelines usually use two frames buffers: one to draw the graphics primitives and one to copy the frame to the display. The SmolPhone does not have enough volatile memory to afford a frame buffer. To meet C1a, we use the scanline rendering algorithm to draw the screen line-by-line. A single line buffer is required. However, we would like to explore multi-line buffer in the future if it makes drawing more efficient.

Some graphics pipelines can have more phases. For example, Flutter adds a compositing phase between the render and draw phases to merge the display list of several rendered layers into a single one.

The Figure 2, Figure 3 and 4 illustrate the graphics pipeline with a to-do list application. The Figure 2 shows the widget tree of the application. It consists of several widgets: three containers, a text field, a button, a text label, a dynamic container, two checkboxes and two texts. The Figure 3 in green the computed bounding boxes of the widgets during the layout phase. Some bounding boxes are omitted in this figure for sake of simplicity (the dynamic container and the two containers enclosing each to-do list item). Finally, the Figure 4 depicts the drawn screen.

Figure 2: Widget Tree of the todo list application.
Figure 3: bounding boxes of the widgets of the todo list application.
Figure 4: Screen of the todo list application.

Currently, the layout phase doesn’t have access to the widgets’ state. This means that a widget with a dynamic content, such as a text widget, cannot be sized based on its content.

The graphics pipeline typically operates in either immediate or retained mode. In immediate mode, the entire pipeline is executed to display each frame. In retained mode, the pipeline is executed entirely to display the first frame, and then only partially for subsequent ones. Most of modern frameworks retain the widget layout tree and re-layout it when needed. They can also retain a part of the display list to efficiently handle screen scrolling. They also save drawing result as textures to speed up the draw phase.

The SmolPhone uses a minimal retained mode that retains the widget layout tree. Only widgets whose state has been updated since the last displayed frame need to be re-rendered. In the previous example, if a user toggles a checkbox, then only this checkbox has to be rendered and then drawn on the screen. In the future, we would like to explore the possibility of retaining part of the display list to efficiently handle screen scrolling. We cannot texturize the drawing results because it could contradict C1a.