Build 10x products in minutes by chatting with AI - beyond just a prototype.
Topics
With the popularity of real-time video call applications skyrocketing in today's interconnected world, developers are constantly looking for seamless and effective solutions to build these applications. If you are working on Flutter, a cross-platform development framework offered by Google, the Twilio Programmable Video package can be your go-to tool. Developed by Twilio, this package allows developers to leverage WebRTC technology to construct real-time videocall applications using a single codebase.
The exciting part is that Twilio Programmable Video usage extends beyond simply adding video calling functionality to your applications. It includes exciting features such as generating access tokens for authentication, managing video streams, and much more. However, please note that this Flutter plugin is a community-maintained project and not maintained by Twilio. For any issues, you should file an issue instead of contacting support.
This blog post will delve into the integration and usage of Twilio Programmable Video in a Flutter application. We will discuss everything from basics to advanced features, starting with installing the Twilio Flutter SDK, generating access tokens, managing video streams, and more.
Before diving into the technical details, let's first understand the basics of Twilio Programmable Video and Flutter.
Twilio Programmable Video is a powerful addition to your developer toolkit. It allows embedding real-time video calling functionality into your applications. It is based on Web Real-Time Communication (WebRTC), an open-source project providing web-based real-time communication.
Twilio's primary feature is the Twilio Video SDK, a robust toolkit for developers to work with video communications on various platforms—from Android to iOS, Web, and in our case, Flutter. The SDK provides the capabilities of managing audio and video streams, creating chat rooms, handling event data, and more, aiding in easy building powerful communication applications.
Flutter, on the other hand, is Google's UI toolkit for natively compiling applications. Known for its powerful hot-reloading feature, Flutter allows the creation of beautiful, natively compiled applications for mobile, web, and desktop from a single codebase.
One of Flutter's many strengths lies in the way it handles packages. Packages in Flutter, like the twilio_programmable_video package we're discussing, allow developers to include code that is either Dart-native or includes code written in other languages, bundled as a library, extending the capabilities of a Flutter app.
Setting up Twilio Programmable Video within a Flutter application involves multiple steps. Before you start, ensure the Flutter SDK is installed on your system. If not, follow Flutter's official guide to install and set up Flutter on your system.
First, we must incorporate the Twilio_programmable_video package into our Flutter app. Below are the steps to set up and install Twilio Programmable Video in your Flutter app:
1dependencies: 2 flutter: 3 sdk: flutter 4 5 twilio_programmable_video: ^latest_version
1$ flutter pub get
Congrats! You have successfully added the Twilio_programmable_video package to your Flutter app. This package allows your Flutter project to utilize the Twilio Video SDK within it, allowing you to add functionalities from creating video rooms to managing participants.
Next, we specify the Android and iOS platform settings, permissions, and configurations required for Twilio Programmable Video.
You must tweak a few files for Android to ensure the package works correctly.
Open the AndroidManifest.xml file in the android/app/src/main directory and add the following device permissions.:
1<uses-permission android:name="android.permission.RECORD_AUDIO" /> 2<uses-permission android:name="android.permission.CAMERA"/> 3<uses-permission android:name="android.permission.BLUETOOTH"/> 4<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
Add the following lines to your android/app/proguard-rules.pro file:
1-keep class tvi.webrtc.** { *; } 2-keep class com.twilio.video.** { *; } 3-keep class com.twilio.common.** { *; } 4-keepattributes InnerClasses
Don't forget to include proguard-rules.pro in your android/app/build.gradle file.
For iOS, you will have to tweak a few files.
Open the Info.plist file in your ios/Runner directory and provide permissions for camera and microphone use:
1<key>NSCameraUsageDescription</key> 2<string>Your message to user when the camera is accessed for the first time</string> 3<key>NSMicrophoneUsageDescription</key> 4<string>Your message to user when the microphone is accessed for the first time</string> 5<key>io.flutter.embedded_views_preview</key> 6<true/>
For the web, add a script tag containing the twilio-video javascript package to index.html in the web directory.
1<!DOCTYPE html> 2<html> 3 <head> 4 ... 5 <script src="//media.twiliocdn.com/sdk/js/video/releases/2.14.0/twilio-video.min.js"></script> 6 </head> 7 .... 8</html>
With this, we successfully set up Twilio Programmable Video with Flutter SDK in your application. You're now ready to start building real-time videocall applications.
In the next sections, we will walk you through the functionality provided by the Twilio Video SDK and how you can use them to build a robust video service.
One crucial aspect of working with any SDK is authentication, which ensures only permitted users use the resources. Twilio Programmable Video utilizes access tokens for this authentication process.
An access token lets a user connect to a specific video room. So, before a user can start using Twilio video, you need to generate an access token either on your client-side app (not recommended for production) or on your server-side, and then pass it onto the Twilio client to start a session.
Now, let's understand how you can generate an access token:
To generate an access token, you need a few details:
Account SID: This is your primary Twilio account identifier - find this in the console here .
API Key & Secret: Create a new API key and secret at the console here , or use the example key pair provided in the Twilio console.
It should be noted that you cannot generate access tokens for programmable video using TestCredentials. Make use of your LIVE credentials.
With these details, you can now generate your access token.
Twilio provides helper libraries in several programming languages to help you generate access tokens on your server-side. Here's a simple example in Python showing how to generate tokens:
1# Import the required package 2from twilio.jwt.access_token import AccessToken 3from twilio.jwt.access_token.grants import VideoGrant 4 5# required for generating a token 6account_sid = 'ACCOUNT_SID' 7api_key = 'API_KEY' 8api_secret = 'API_SECRET' 9 10# Create an Access Token 11token = AccessToken(account_sid, api_key, api_secret, identity='example-user') 12 13# Create a Video Grant and add to token 14video_grant = VideoGrant(room='cool room') 15token.add_grant(video_grant) 16 17# Serialize the token as a JWT 18jwt = token.to_jwt() 19print(jwt)
This token will then connect to a video room from your Flutter app.
Now that you know how to generate access tokens, the next step is validating these tokens and allowing users to connect to a video room.
Now that you have generated your access tokens, you can implement the video calling functionality in your Flutter app. First, you will have to connect to a room.
To connect to a Room in your Flutter app, use the TwilioProgrammableVideo.connect() method. After successfully connecting, you can send and receive voice and video broadcasts with other Participants in the Room.
Here's how you do it:
1import 'package:twilio_programmable_video/twilio_programmable_video.dart'; 2 3Room _room; 4final Completer<Room> _completer = Completer<Room>(); 5 6void _onConnected(Room room) { 7 print('Connected to ${room.name}'); 8 _completer.complete(_room); 9} 10 11void _onConnectFailure(RoomConnectFailureEvent event) { 12 print('Failed to connect to room ${event.room.name} with exception: ${event.exception}'); 13 _completer.completeError(event.exception); 14} 15 16Future<Room> connectToRoom() async { 17 var connectOptions = ConnectOptions( 18 "access_token_here", // replace with your generated access token 19 roomName: "roomName", // specify your room name 20 // Other optional parameters 21 ); 22 _room = await TwilioProgrammableVideo.connect(connectOptions); 23 _room.onConnected.listen(_onConnected); 24 _room.onConnectFailure.listen(_onConnectFailure); 25 return _completer.future; 26}
You must pass the Access Token when connecting to a Room. The ConnectOptions class has many properties that you can use to configure your video call according to your requirements.
Once you have connected to the room, the next step is to display the video from your camera and microphone to your app.
Here's how you can do it:
1// Create an audio track. 2var localAudioTrack = LocalAudioTrack(true); 3 4// Retrieve the camera source of your choosing 5var cameraSources = await CameraSource.getSources(); 6var cameraCapturer = CameraCapturer( 7 cameraSources.firstWhere((source) => source.isFrontFacing), 8); 9 10// Create a video track. 11var localVideoTrack = LocalVideoTrack(true, cameraCapturer); 12 13// Getting the local video track widget. 14// This can only be called after the TwilioProgrammableVideo.connect() is called. 15var widget = localVideoTrack.widget();
This section covers how to integrate the video call functionality into your Flutter app, from connecting to a room to displaying the video from your device.
In the next section, we will dive into some of the advanced features and capabilities Twilio Programmable Video provides.
Twilio Programmable Video offers many advanced features that give you granular control over your video conferencing application. Let's have a look at some of them:
Twilio Programmable Video gives the ability to manage and customize video streams according to your needs. For example, you can have multiple streams in a single room, allowing multiple participants to share their video feed. Each stream can be controlled and customized - you can mute/unmute audio, enable/disable video, and much more:
1var localAudioTrack = LocalAudioTrack(true); 2localAudioTrack.enable(false); // Mutes the audio track 3 4var localVideoTrack = LocalVideoTrack(true, await CameraCapturer(CameraSource.frontCamera)); // "frontCamera" captures video from your front camera 5localVideoTrack.enable(false); // Disables the video track
Twilio Programmable Video lets you handle remote participants in real-time. Upon connecting to a room, you may find some participants that are already present. When you use the Room, you can check for existing participants by calling the Room.onConnected listener.remoteParticipant getter:
1// Connect to a room. 2var room = await TwilioProgrammableVideo.connect(connectOptions); 3 4room.onConnected((Room room) { 5 // After receiving the connected callback the LocalParticipant becomes available. 6 var localParticipant = room.localParticipant; 7 8 // Get the first participant from the room. 9 var remoteParticipant = room.remoteParticipants[0]; 10});
When participants connect to or disconnect from a Room you are connected to, you will be notified via an event listener, allowing your application to keep track of who joins and leaves the Room.
1// Connect to a room. 2var room = await TwilioProgrammableVideo.connect(connectOptions); 3 4room.onParticipantConnected((RoomParticipantConnectedEvent event) { 5 print('Participant ${event.remoteParticipant.identity} has joined the room'); 6}); 7 8room.onParticipantDisconnected((RoomParticipantDisconnectedEvent event) { 9 print('Participant ${event.remoteParticipant.identity} has left the room'); 10});
Twilio Programmable Video's API allows you to construct a DataTrack channel that sends low latency messages to all subscribing receivers. It would be best if you mentioned it in the ConnectOptions when connecting to a room:
1var connectOptions = ConnectOptions( 2 "access_token", // your access token here 3 // Other options 4 dataTracks: [ 5 LocalDataTrack( 6 DataTrackOptions( 7 ordered: true, 8 // Other options 9 ), 10 ), 11 ], 12);
Using such rich and robust features of the Twilio Programmable Video, you can easily create a feature-filled, secure, and performative video calling application using Flutter.
As with any technology or package, developers may face some common issues while working with Twilio Programmable Video in Flutter. Awareness of these problems and their respective solutions can save much debugging time. Let's discuss some typical problems:
Access tokens are paramount for authentication to connect to a Room. If your access token is improperly generated or invalid, you will encounter an AccessTokenError exception.
Solution: Ensure that you are generating your tokens correctly. Always verify your Account SID, API Key, and API Secret while generating access tokens. Also, you cannot use test credentials for generating tokens; only LIVE credentials should be used.
Sometimes, you might find that there's no video or audio in the room upon successful connection.
Solution: Ensure you have given the necessary permissions to access the camera and microphone. Check your app's permission settings to guarantee they are enabled.
This issue generally arises due to platform-specific settings. For instance, if you have configured the Android settings but overlooked the iOS settings, your app might work on Android but not on iOS and vice versa.
Solution: Ensure you have made the necessary platform-specific configurations in Android and iOS settings.
You might run into a null error when you try to get a widget from the local video track right after making a connection.
Solution: The LocalVideoTrack.widget() method can only be called after TwilioProgrammableVideo.connect() is called. So, ensure you get the widget once the connection is made.
Real-time video applications have become paramount in today’s digital world and Flutter with Twilio Programmable Video provides a powerful and efficient way to build such applications. By leveraging Twilio’s robust features such as access token-based authentication, managing video streams, and handling remote participants and events, you can take your Flutter application to new heights of user engagement.
In this blog post, we introduced how to integrate Twilio Programable Video in your Flutter app, discussed the basics, and showed how to connect to a Room, and displayed video. We also discussed some common issues and their solutions when working with Twilio and Flutter.
Remember, building real-time video apps goes beyond the technical implementation. The User Experience is equally critical. Ensure your app offers a smooth and seamless video experience, as that is what your users will ultimately interact with.
And always, keep fluttering!
