Building Real-Time Videocall Applications with Twilio Programmable Video in Flutter

With the popularity of real-time video call applications skyrocketing in today's interconnected world, developers are constantly looking for seamless and effective solutions to build these applications. If you are working on Flutter, a cross-platform development framework offered by Google, the Twilio Programmable Video package can be your go-to tool. Developed by Twilio, this package allows developers to leverage WebRTC technology to construct real-time videocall applications using a single codebase.

The exciting part is that Twilio Programmable Video usage extends beyond simply adding video calling functionality to your applications. It includes exciting features such as generating access tokens for authentication, managing video streams, and much more. However, please note that this Flutter plugin is a community-maintained project and not maintained by Twilio. For any issues, you should file an issue instead of contacting support.

This blog post will delve into the integration and usage of Twilio Programmable Video in a Flutter application. We will discuss everything from basics to advanced features, starting with installing the Twilio Flutter SDK, generating access tokens, managing video streams, and more.

Basics of Twilio Programmable Video

Before diving into the technical details, let's first understand the basics of Twilio Programmable Video and Flutter.

Twilio Programmable Video is a powerful addition to your developer toolkit. It allows embedding real-time video calling functionality into your applications. It is based on Web Real-Time Communication (WebRTC), an open-source project providing web-based real-time communication.

Twilio's primary feature is the Twilio Video SDK, a robust toolkit for developers to work with video communications on various platforms—from Android to iOS, Web, and in our case, Flutter. The SDK provides the capabilities of managing audio and video streams, creating chat rooms, handling event data, and more, aiding in easy building powerful communication applications.

Flutter, on the other hand, is Google's UI toolkit for natively compiling applications. Known for its powerful hot-reloading feature, Flutter allows the creation of beautiful, natively compiled applications for mobile, web, and desktop from a single codebase.

One of Flutter's many strengths lies in the way it handles packages. Packages in Flutter, like the twilio_programmable_video package we're discussing, allow developers to include code that is either Dart-native or includes code written in other languages, bundled as a library, extending the capabilities of a Flutter app.

Getting Started with Twilio Programmable Video in Flutter

Setting up Twilio Programmable Video within a Flutter application involves multiple steps. Before you start, ensure the Flutter SDK is installed on your system. If not, follow Flutter's official guide to install and set up Flutter on your system.

First, we must incorporate the Twilio_programmable_video package into our Flutter app. Below are the steps to set up and install Twilio Programmable Video in your Flutter app:

Add twilio_programmable_video in pubspec.yaml:

• Add the following line in the dependencies section of pubspec.yaml:

Copy
1
2
3
4
5
dependencies:
  flutter:
    sdk: flutter

  twilio_programmable_video: ^latest_version

• Run the following command in the terminal to get packages:

Copy
1
$ flutter pub get

Congrats! You have successfully added the Twilio_programmable_video package to your Flutter app. This package allows your Flutter project to utilize the Twilio Video SDK within it, allowing you to add functionalities from creating video rooms to managing participants.

Next, we specify the Android and iOS platform settings, permissions, and configurations required for Twilio Programmable Video.

Android Setup:

You must tweak a few files for Android to ensure the package works correctly.

Permissions:

Open the AndroidManifest.xml file in the android/app/src/main directory and add the following device permissions.:

Copy
1
2
3
4
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA"/>
<uses-permission android:name="android.permission.BLUETOOTH"/>
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>

Proguard:

Add the following lines to your android/app/proguard-rules.pro file:

Copy
1
2
3
4
-keep class tvi.webrtc.** { *; }
-keep class com.twilio.video.** { *; }
-keep class com.twilio.common.** { *; }
-keepattributes InnerClasses

Don't forget to include proguard-rules.pro in your android/app/build.gradle file.

iOS Setup:

For iOS, you will have to tweak a few files.

Permissions:

Open the Info.plist file in your ios/Runner directory and provide permissions for camera and microphone use:

Copy
1
2
3
4
5
6
<key>NSCameraUsageDescription</key>
<string>Your message to user when the camera is accessed for the first time</string>
<key>NSMicrophoneUsageDescription</key>
<string>Your message to user when the microphone is accessed for the first time</string>
<key>io.flutter.embedded_views_preview</key>
<true/>

Web Setup:

For the web, add a script tag containing the twilio-video javascript package to index.html in the web directory.

Copy
1
2
3
4
5
6
7
8
<!DOCTYPE html>
<html>
  <head>
    ...
    <script src="//media.twiliocdn.com/sdk/js/video/releases/2.14.0/twilio-video.min.js"></script>
  </head>
  ....
</html>

With this, we successfully set up Twilio Programmable Video with Flutter SDK in your application. You're now ready to start building real-time videocall applications.

In the next sections, we will walk you through the functionality provided by the Twilio Video SDK and how you can use them to build a robust video service.

Authentication with Twilio Programmable Video

One crucial aspect of working with any SDK is authentication, which ensures only permitted users use the resources. Twilio Programmable Video utilizes access tokens for this authentication process.

An access token lets a user connect to a specific video room. So, before a user can start using Twilio video, you need to generate an access token either on your client-side app (not recommended for production) or on your server-side, and then pass it onto the Twilio client to start a session.

Now, let's understand how you can generate an access token:

Generating an Access Token:

To generate an access token, you need a few details:

• Account SID: This is your primary Twilio account identifier - find this in the console here .

• API Key & Secret: Create a new API key and secret at the console here , or use the example key pair provided in the Twilio console.

It should be noted that you cannot generate access tokens for programmable video using TestCredentials. Make use of your LIVE credentials.

With these details, you can now generate your access token.

Twilio provides helper libraries in several programming languages to help you generate access tokens on your server-side. Here's a simple example in Python showing how to generate tokens:

Copy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Import the required package
from twilio.jwt.access_token import AccessToken
from twilio.jwt.access_token.grants import VideoGrant

# required for generating a token
account_sid = 'ACCOUNT_SID'
api_key = 'API_KEY'
api_secret = 'API_SECRET'

# Create an Access Token
token = AccessToken(account_sid, api_key, api_secret, identity='example-user')

# Create a Video Grant and add to token
video_grant = VideoGrant(room='cool room')
token.add_grant(video_grant)

# Serialize the token as a JWT
jwt = token.to_jwt()
print(jwt)

This token will then connect to a video room from your Flutter app.

Now that you know how to generate access tokens, the next step is validating these tokens and allowing users to connect to a video room.

Embedding Twilio Videos in Your Flutter App

Now that you have generated your access tokens, you can implement the video calling functionality in your Flutter app. First, you will have to connect to a room.

Connecting to a Room:

To connect to a Room in your Flutter app, use the TwilioProgrammableVideo.connect() method. After successfully connecting, you can send and receive voice and video broadcasts with other Participants in the Room.

Here's how you do it:

Copy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import 'package:twilio_programmable_video/twilio_programmable_video.dart';

Room _room;
final Completer<Room> _completer = Completer<Room>();

void _onConnected(Room room) {
  print('Connected to ${room.name}');
  _completer.complete(_room);
}

void _onConnectFailure(RoomConnectFailureEvent event) {
  print('Failed to connect to room ${event.room.name} with exception: ${event.exception}');
  _completer.completeError(event.exception);
}

Future<Room> connectToRoom() async {
  var connectOptions = ConnectOptions(
    "access_token_here",  // replace with your generated access token
    roomName: "roomName", // specify your room name
    // Other optional parameters
  );
  _room = await TwilioProgrammableVideo.connect(connectOptions);
  _room.onConnected.listen(_onConnected);
  _room.onConnectFailure.listen(_onConnectFailure);
  return _completer.future;
}

You must pass the Access Token when connecting to a Room. The ConnectOptions class has many properties that you can use to configure your video call according to your requirements.

Display the Video:

Once you have connected to the room, the next step is to display the video from your camera and microphone to your app.

Here's how you can do it:

Copy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Create an audio track.
var localAudioTrack = LocalAudioTrack(true);

// Retrieve the camera source of your choosing
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
  cameraSources.firstWhere((source) => source.isFrontFacing),
);

// Create a video track.
var localVideoTrack = LocalVideoTrack(true, cameraCapturer);

// Getting the local video track widget.
// This can only be called after the TwilioProgrammableVideo.connect() is called.
var widget = localVideoTrack.widget();

This section covers how to integrate the video call functionality into your Flutter app, from connecting to a room to displaying the video from your device.

In the next section, we will dive into some of the advanced features and capabilities Twilio Programmable Video provides.

Advanced Features of Twilio Programmable Video

Twilio Programmable Video offers many advanced features that give you granular control over your video conferencing application. Let's have a look at some of them:

Managing and Customizing Video Streams

Twilio Programmable Video gives the ability to manage and customize video streams according to your needs. For example, you can have multiple streams in a single room, allowing multiple participants to share their video feed. Each stream can be controlled and customized - you can mute/unmute audio, enable/disable video, and much more:

Copy
1
2
3
4
5
var localAudioTrack = LocalAudioTrack(true);
localAudioTrack.enable(false); // Mutes the audio track

var localVideoTrack = LocalVideoTrack(true, await CameraCapturer(CameraSource.frontCamera)); // "frontCamera" captures video from your front camera
localVideoTrack.enable(false); // Disables the video track

Handling Remote Participants:

Twilio Programmable Video lets you handle remote participants in real-time. Upon connecting to a room, you may find some participants that are already present. When you use the Room, you can check for existing participants by calling the Room.onConnected listener.remoteParticipant getter:

Copy
1
2
3
4
5
6
7
8
9
10
// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onConnected((Room room) {
  // After receiving the connected callback the LocalParticipant becomes available.
  var localParticipant = room.localParticipant;

  // Get the first participant from the room.
  var remoteParticipant = room.remoteParticipants[0];
});

When participants connect to or disconnect from a Room you are connected to, you will be notified via an event listener, allowing your application to keep track of who joins and leaves the Room.

Copy
1
2
3
4
5
6
7
8
9
10
// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onParticipantConnected((RoomParticipantConnectedEvent event) {
  print('Participant ${event.remoteParticipant.identity} has joined the room');
});

room.onParticipantDisconnected((RoomParticipantDisconnectedEvent event) {
  print('Participant ${event.remoteParticipant.identity} has left the room');
});

Using the DataTrack API:

Twilio Programmable Video's API allows you to construct a DataTrack channel that sends low latency messages to all subscribing receivers. It would be best if you mentioned it in the ConnectOptions when connecting to a room:

Copy
1
2
3
4
5
6
7
8
9
10
11
12
var connectOptions = ConnectOptions(
  "access_token", // your access token here
  // Other options
  dataTracks: [
    LocalDataTrack(
      DataTrackOptions(
        ordered: true,
        // Other options
      ),
    ),
  ],
);

Using such rich and robust features of the Twilio Programmable Video, you can easily create a feature-filled, secure, and performative video calling application using Flutter.

Common Issues with Twilio Programmable Video and their Solutions

As with any technology or package, developers may face some common issues while working with Twilio Programmable Video in Flutter. Awareness of these problems and their respective solutions can save much debugging time. Let's discuss some typical problems:

Issue 1: Cannot Connect to Room due to Invalid Access Token

Access tokens are paramount for authentication to connect to a Room. If your access token is improperly generated or invalid, you will encounter an AccessTokenError exception.

Solution: Ensure that you are generating your tokens correctly. Always verify your Account SID, API Key, and API Secret while generating access tokens. Also, you cannot use test credentials for generating tokens; only LIVE credentials should be used.

Issue 2: No Video or Audio in the Room

Sometimes, you might find that there's no video or audio in the room upon successful connection.

Solution: Ensure you have given the necessary permissions to access the camera and microphone. Check your app's permission settings to guarantee they are enabled.

Issue 3: App Works on One Platform but Not on Another

This issue generally arises due to platform-specific settings. For instance, if you have configured the Android settings but overlooked the iOS settings, your app might work on Android but not on iOS and vice versa.

Solution: Ensure you have made the necessary platform-specific configurations in Android and iOS settings.

Issue 4: Null Error when Trying to Display Video

You might run into a null error when you try to get a widget from the local video track right after making a connection.

Solution: The LocalVideoTrack.widget() method can only be called after TwilioProgrammableVideo.connect() is called. So, ensure you get the widget once the connection is made.

Conclusion

Real-time video applications have become paramount in today’s digital world and Flutter with Twilio Programmable Video provides a powerful and efficient way to build such applications. By leveraging Twilio’s robust features such as access token-based authentication, managing video streams, and handling remote participants and events, you can take your Flutter application to new heights of user engagement.

In this blog post, we introduced how to integrate Twilio Programable Video in your Flutter app, discussed the basics, and showed how to connect to a Room, and displayed video. We also discussed some common issues and their solutions when working with Twilio and Flutter.

Remember, building real-time video apps goes beyond the technical implementation. The User Experience is equally critical. Ensure your app offers a smooth and seamless video experience, as that is what your users will ultimately interact with.

And always, keep fluttering!