Build 10x products in minutes by chatting with AI - beyond just a prototype.
Topics
Welcome to our new journey in the world of Flutter and GLSL, where we'll be discussing in detail about shaders. So, what is a shader? Shaders, including both fragment shaders and vertex shaders, are small programs that run on a Graphics Processing Unit (GPU) that manipulate the attributes of either pixel (also known as fragments) or vertices, the primary constructs of 3D graphics. Designed in the OpenGL shading language (GLSL), shaders define how the pixels and vertices should be rendered on the screen, powering the visuals you see in modern browsers.
Let's break this down further. The vertex shader operates first and foremost in the shader stage of the graphics pipeline. It processes the vertex data provided by the previous stages of the graphics pipeline and computes the final vertex positions. On the other hand, a fragment shader, also known as a pixel shader, processes fragment colors, a pixel's color value and its depth value, before the final image is placed on the screen.
The importance of shaders, particularly in Graphics and App development, truly cannot be overstated. They offer the creative and processing power needed to define the look and feel of 3D graphics, making them an integral part of Flutter app development.
Stay with me as we dive deeper into the world of fragment shaders. But before proceeding to the next section, let's clarify the differences and similarities between fragment shaders and vertex shaders.
As mentioned earlier, vertex shaders are focused on processing the vertex data in the graphics pipeline while fragment shaders (or pixel shaders) work on fragment colors and depth values. But when can we choose one over the other?
In simple terms, if your operations heavily depend on the vertex position and other attributes, a vertex shader is your go-to. On the other hand, operations that depend on determining the color output of the rendering process are better suited for fragment shaders.
It's also worth noting that sometimes, the operations of fragment shaders and vertex shaders can work in tandem. For instance, a vertex shader might compute the vertex position of an image, which the fragment shader later uses to generate the pixel color data.
Congratulations on making it to this stage! Now that we have a solid understanding of what a shader is, let's explore the world of fragment shaders in Flutter and GLSL (OpenGL Shading Language).
Flutter, as we know, is an open-source UI software development toolkit, created by Google. It has gained immense popularity for enabling developers to craft beautiful, natively compiled applications for mobile, web, and desktop—all from a single codebase.
GLSL, on the other hand, is short for OpenGL Shading Language. It is the language that allows us to write shaders for the OpenGL graphics API, offering the flexibility to directly control the graphics pipeline. In the context of this blog, we will focus on GLSL fragment shaders.
A fragment shader's role in a Flutter app is to define how the color values and depth value of each pixel should be computed, given specific lighting conditions, texture coordinates, and other factors.
Flutter doesn't currently support vertex shaders, but it generously supports fragment shaders, enabling developers to enhance the visual aesthetics of their Flutter apps. By convention, these GLSL fragment shaders have a .frag extension.
Great! Now that we have grasped an understanding of how fragment shaders work, let's get down to adding them to a Flutter project. Flutter allows the addition of custom shaders in the form of GLSL files with the .frag extension.
The first step in leveraging the power of fragment shaders in your Flutter apps is by including them in your Flutter project. You can do this by naming them within the shaders section of your project’s pubspec.yaml file. Here's a code snippet illustrating the concept:
1 flutter: 2 shaders: 3 - shaders/myshader.frag 4
The Flutter command-line tool will compile the shader to its proper backend, transforming your GLSL fragment shader code into the appropriate format for either the WebGL, OpenGL, or Vulkan backend.
Post-declaration of the shaders, the Flutter tooling will compile them to their appropriate backend format, and generate the necessary runtime metadata. This allows the shaders to be fully compatible with your chosen graphic interface and for the fragment shader to be included in the application.
After successful compilation and runtime metadata generation, the shader is then included in the application, just like an asset.
Moving on to the next exciting part of fragment shaders, we will explore how shaders can be dynamically compiled in Flutter to offer flexible, at-runtime graphics rendering in your applications.
One of the major benefits of using Flutter is the hot reload and hot restart feature. Due to this innovative feature, if you make any changes to your shader programs, it triggers a shader recompilation. This live recompilation is not only limited to fragment shaders but extends to all custom shaders. A hot reload or restart updates your shader without having to manually stop and start your app. This dynamic compilation feature saves precious developer time and significantly hastens the development process.
Another noteworthy aspect is that when running your application in debug mode, changes made to a shader program trigger recompilation. This way, the updated version of your shader (vertex or fragment) is executed during the next hot reload or hot restart operation, ensuring that you always work with the latest version of your shader.
Loading shaders during runtime can bring a lot of flexibility to your Flutter application. Let's see how it's done in Flutter using the FragmentProgram API.
In Flutter, to load a shader into a FragmentProgram object during runtime, you would use the FragmentProgram.fromAsset constructor. This constructor is asynchronous and must be awaited. The asset’s name is the same as the path to the shader given in the pubspec.yaml file. Here's an example:
1 void loadMyShader() async { 2 var program = await FragmentProgram.fromAsset('shaders/myshader.frag'); 3 } 4
This function loads the shader named 'myshader.frag' from the shaders directory. After this step, your shader is ready to use where you need it in your Flutter application.
Now that we have loaded our shaders, let's discover how to effectively deal with the FragmentShader instances in your Flutter application.
The FragmentProgram object, which we have already worked with while loading the shader, can be further used to create one or more FragmentShader instances. A FragmentShader represents a fragment program along with a particular set of uniforms or configuration parameters.
Here's how you can create an instance of FragmentShader:
1 void updateShader(Canvas canvas, Rect rect, FragmentProgram program) { 2 var shader = program.fragmentShader(); 3 shader.setFloat(0, 42.0); 4 canvas.drawRect(rect, Paint()..shader = shader); 5 } 6
Uniforms are essentially variables that are global to a program object, which includes both a vertex and a fragment shader. They can be effectively used to control shader behavior.
In the context of fragment shaders, the set of uniforms defined serves as the input parameters to the fragment shader. These can be set using the setFloat and setImageSampler methods and vary depending on how the shader was defined.
Authoring fragment shaders can be a thrilling experience. By creating your own custom shaders, you can achieve the exact visuals you envision for your Flutter application.
Fragment shaders are authored using GLSL source files. By convention, these files bear the .frag extension in the context of Flutter. Unlike some other platforms, Flutter doesn't currently support vertex shaders, which would typically have the .vert extension.
The Flutter framework supports any GLSL version from 460 down to 100, though some features are restricted based on the platform and version. Importantly, developers should be aware of the limitations when authoring shaders for Flutter. For instance, Unsigned integers and booleans aren't currently supported, and all precision hints are ignored when targeting Skia.
UBOs (Uniform Buffer Objects) and SSBOs (Shader Storage Buffer Objects) are advanced GLSL features that allow for efficient storage and retrieval of large amounts of data on the GPU. While extraordinarily powerful, they're currently not supported in Flutter due to certain constraints at the Skia engine level. Instead, Flutter relies on simpler alternatives - uniform variables.
When defining fragment shaders, special variables known as uniforms control the behavior of the shader. Let's delve into these and understand how to work with uniforms in fragment shaders effectively.
GLSL uniforms are defined in the shader source and then set in Dart for each fragment shader instance. These uniforms, in essence, act as the configuration parameters altering the resultant shader output, thereby providing a lot of flexibility while working with fragment shaders.
There are various types of uniforms you can work with, including floating-point uniforms defined with GLSL types float, vec2, vec3, and vec4. These are set using the FragmentShader.setFloat method. Additionally, GLSL sampler values that use the sampler2D type, are set using the FragmentShader.setImageSampler method.
To demonstrate, let's say we have the following uniforms declared in a GLSL fragment program:
1 uniform float uScale; 2 uniform sampler2D uTexture; 3 uniform vec2 uMagnitude; 4 uniform vec4 uColor; 5
In Dart, the equivalent code for initializing these uniform values would look like this:
1 void updateShader(FragmentShader shader, Color color, Image image) { 2 shader.setFloat(0, 23); // uScale 3 shader.setFloat(1, 114); // uMagnitude x 4 shader.setFloat(2, 83); // uMagnitude y 5 6 // Convert color to premultiplied opacity. 7 shader.setFloat(3, color.red / 255 * color.opacity); // uColor r 8 shader.setFloat(4, color.green / 255 * color.opacity); // uColor g 9 shader.setFloat(5, color.blue / 255 * color.opacity); // uColor b 10 shader.setFloat(6, color.opacity); // uColor a 11 12 // Initialize sampler uniform. 13 shader.setImageSampler(0, image); 14 } 15
It's important to note that the indices used with FragmentShader.setFloat do not count the sampler2D uniform. Furthermore, any unidentified float uniforms, or those that haven't been set explicitly, will default to 0.0.
When authoring fragment shaders, it is prevalent that the shader's output depends on the current position of the pixel (or fragment). Let's see how Flutter provides access to these local coordinates.
In Flutter, the shader has access to a varying value that contains the local coordinates for the specific fragment being evaluated. This feature is vital to compute effects that depend on the current position.
Import the flutter/runtime_effect.glsl library and call the FlutterFragCoord function to access it. Here's an example:
1 #include <flutter/runtime_effect.glsl> 2 3 void main() { 4 vec2 currentPos = FlutterFragCoord().xy; 5 } 6
The flutter/runtime_effect.glsl library is a collection of fragment shader utility functions provided by Flutter to ease the process of writing GLSL code for use in your Flutter application. It includes several utility macros and functions, one of which is FlutterFragCoord which provides the current fragment’s position.
The value extracted from FlutterFragCoord is quite different from gl_FragCoord. While gl_FragCoord provides the screen space coordinates, FlutterFragCoord offers the local coordinates within a given render object. gl_FragCoord should generally be avoided to ensure that shaders are consistent across different backends.
Colors play a crucial role in shaders. In this segment, we'll discuss how colors are usually represented in fragment shaders and understand the concept of samplers.
In shaders, there isn't a built-in data type for colors. Instead, colors are typically represented as a vec4 with each component corresponding to one of the RGBA color channels.
The color output, referred to as fragColor, is expected to range from 0.0 to 1.0 and should have premultiplied alpha. This is distinct from typical Flutter colors, which use a 0-255 value encoding and have unpremultiplied alpha.
A sampler in a fragment shader provides access to a dart:ui Image object. This image object can be sourced from a decoded image or from part of the application using Scene.toImageSync or Picture.toImageSync.
Here is an example of how you can access textures using samplers:
1 #include <flutter/runtime_effect.glsl> 2 3 uniform vec2 uSize; 4 uniform sampler2D uTexture; 5 6 out vec4 fragColor; 7 8 void main() { 9 vec2 uv = FlutterFragCoord().xy / uSize; 10 fragColor = texture(uTexture, uv); 11 } 12
Designing visually pleasing applications using fragment shaders is important, but keeping performance in mind is crucial. Let's discuss a few considerations to optimize the performance of your Flutter applications while using fragment shaders.
Shader loading, notably when targeting the Skia backend, maybe resource-intensive since it necessitates the shader's compilation to a platform-specific shader at runtime. The Flutter command-line tool handles this, but it's an operation that could be avoided with diligent planning.
One of the ways to lessen the load and improve the performance is to pre-cache the fragment program objects before initiating an animation that leverages one or more shaders. By loading the shaders beforehand, you can prevent unnecessary delays during the animation, leading to a much smoother experience.
Another bit of good news is that you can reuse a FragmentShader object across frames, which is far more efficient than creating a new FragmentShader for each frame. This reuse can significantly improve the rendering performance of your animations.
Performance considerations mark an important part of working with fragment shaders. Being aware of these tips and tricks will certainly help you create not only beautiful but also efficient Flutter applications.
Through this blog post, we have undertaken an exciting journey involving the world of shaders, more explicitly fragment shaders, and the role they play in crafting visually stunning Flutter applications. We have comprehended what a shader is, explored the key differences and applications of both vertex and fragment shaders, and the great flexibility provided by fragment shaders when loaded during runtime in Flutter.
There's no denying that understanding and implementing fragment shaders is an advanced topic in Flutter app development. However, by harnessing this power, Flutter developers can create applications with rich graphical effects extending beyond the features provided by the Flutter SDK alone.
But don't just stop here! Your exploration of shaders doesn't have to end with this blog post. I encourage you to play around with fragment shaders, experiment with the GLSL code, and come up with something truly unique for your next Flutter project.
Keep fluttering and happy coding!
