Sunday, October 31, 2010

RTMP Part 8 - Packets Examples

This part, I will show a packet trace and description I had done on Red 5 for streaming a vlc file.

Below is the complete packets list from a remote browser connecting to a Red 5 olfaDemo streaming server. This demo provide vlc publishing and streaming service. As you can see, it complete 3 parts of handshakes before the first invoke command. I will ignore handshake in this discussion. Also, please note that bytes are represented in hexadecimal format

Command Message

It is used to perform RPC in the server. In Wireshark, it is named as Invoke message.

I will go through the RTMP header for this packets

03 00 00 7F 00 00 2F 14 00 00 00 00

03 - It is the FMT 0 and Chunk Stream Id 3.

FMT 0 means it is use Chunk Type 0 of 11 bytes

Chunk Stream Id 3 means this AMF object identification number is 3

Since it is Type 0 chunk, the remaining header means

00 00 7F - timestamp delta

00 00 2F - message length

14 - message type id. It is 0x14 (20) . Message type value 20 means it is a AMF 0 encoding command message. Please note that if it is 0x11 (17), it means it is a AMF 3 encoding command message

00 00 00 00 - message stream id

Now, look at the RTMP body, it starts with 02 00 22 64 65 6D ....... 00 40 00 00 00 00 00 00 00 05

This is a encoded AMF 0 message, I will just explain it meaning

First byte is 0x02 (String marker). It follows by 0x 00 22 (String length of 34 bytes). Then, 34 bytes of UTF-8 String

After 32 bytes of string data, the next byte is 0 (Number marker). It following by 8 bytes of IEEE-754 double precision floating point value in network byte order (big-endian).

The message end with 05 (Null marker)

Below show a reply from the server after an Command Message (Invoke).

Header and body information is the same format. Most important note is the first AMF message is a string with value _result

CreateStream Command

createStream command is called using Invoke function call of message type id 0x14 (AMF 0 Command Message).

As you can see, the RTMP header is the same while the RTMP body start with a string marker with createStream as value. Please note that the value after the createStream are argument for the RPC

Play Command

play command is sent as 0x14 message type id (AMF 0 Command Message) as well. This example is clearer to show that the AMF message is asking the server to play (command) ironman2.flv (argument)

Chunk Size Command

After play command, you can see that the server send the chunk size to client.

It is a RTMP protocol command. Let go through the header 42 00 00 00 00 00 04 01

42 - this show FMT as 1 and Chunk Stream Id as 2

FMT 1 refers to Type 1 of Chunk Message header

As this is a RTMP protocol message, the Chunk Stream Id must be 2

FMT 1 tells us that the header is 7 bytes, so, the remaining header is

00 00 00 - timestamp delta

00 00 04 - message length

01 - message type id (this is a Set Chunk Size message)

Since it is a Set Chunk Size message, it have a payload of 4 bytes, that is, 00 00 04 00

User Control Message

As you can see, the server and client send Ping message to one another occasionally. Please note that Wireshark named it as Ping message, however, it is just a User Control Message and may not mean "Ping". You have to look at the content of the payload for the real meaning.

The header is 02 00 00 00 00 00 06 04 00 00 00 00.

02 - It is a FMT is 0 and Chunk Stream id is 02

This tells you it is using 11 bytes chunk message header and it is a protocol message. So, the remaining header is

00 00 00 - timestamp delta

00 00 06 - message length

04 - message type id. This show that it is a User Control Message

00 00 00 00 - message stream id

Since it is a User Control Message, the payload has a format of 2 bytes event type and follow event data. The payload is 00 04 00 00 00 01

00 04 - This mean server is telling client that this is a recorded stream (StreamIsRecorded). It has 4 bytes of event data

00 00 00 01 - This is the stream ID of the recorded stream.

Please refer to previosus RTMP discussion for the event id and meaning. Also, please see the message flow discussion

Data Message

The server is now sending Metadata to the client with Data Message. In Wireshark, it named it as Notify message

The header is 04 00 00 00 00 00 F4 12 01 00 00 00

04 - FMT is 0 and Chunk Stream Id is 4. Thus, it has a chunk stream header size of 11 bytes

00 00 00 - timestamp delta

00 00 F4 - message length

12 - message type id. 0x12 (18) means this is a Data Message using AMF 0 encoding. If it is 0x0F (15), it is a Data Message using AMF 3 encoding

01 00 00 00 - message stream id. The message stream id is 1 because it is stored in little-endian format

Another Note is that message stream id 0x 00 00 00 00 (Integer 0) is reserved for NetConnection Object while 0x 01 00 00 00 (Integer 1) is reserved for NetStream.

I will not go through onMetaData here. These are the decoding audio/video configuration information. It will be needed to initialize the decoder for decoding process.

Audio Data

As you can see, it has a message type id (function call) of 0x08

Incoming Video Data

As you can see, it has a message type id (function call) of 0x09

Friday, October 29, 2010

RTMP Part 7 - Message Example

This part show some example of RTMP message exchange.

Publish Recorded Video

Broadcasting a Share Object Message

Publish MetaData From Recorded Stream

RTMP Part 6 - NetStream Command

It is used to define channel through which the streaming video, audio and data can flow over NetConnection that connect the client to server

A NetConnection can support multiple NetStream for multiple data stream.

1. play

The client tell the server to play a stream. Call play more than once with reset=false, it create a playlist. To play a stream immediately, send a play command with reset=true.

Command structure from client to server

Command structure from server to client
Message flow for play command

2. play2

Unlike play command, it should be used to switch stream bit rate without changing the timeline of the content played.

Command structure for play2 command

Message flow for play2 command

3. deleteStream

It is sent when NetStream is being destory

Command structure for deleteStream. Server will not send any response

4. receiveAudio

It is sent to tell server whether to send audio message to client

Command structure for receiveAudio. Server will not send any response

5. receiveVideo

It is sent to tell server whether to send video message to client

Command structure for receiveVideo. Server will not send any response

6. publish

It is used to publish a named stream to server. Any other client can play this stream with the published name

Command structure for publish command. The server response with onStatus command to mark beginning of publishing

7. seek

It is sent when seeking for offset in millisecond within a media file.

Command structure for seek command. The server send a status message NetStream.Seek.Notify when seek is successful. In failure, server sends _error message
8. pause

It is used to tell server to pause or start a stream

Command structure for pause command. The server send a status message NetStream.Pause.Notify when stream is paused. The server send a status message NetStream.Unpause.Notify when stream is unpaused. In failure, server sends _error message

RTMP Part 5 - NetConnection Command

This part will go deeper detail to NetConnection command.

NetConnection manages communcations between server and client. It also provide asynchronous RPC. It supports the following command

1. connect

It is used to request connection from client to server.

The command has the following structure

Within command object, name-value pair can be inserted as information

Audio codec properties

Video codec properties

Video function properties

Object encoding properties

Command structure from server to client

Below is the command flow for connect command
2. call

It is used to run RPC at the receiver

The structure from sender to receiver is as follow

The response structure

3. createStream

It is used to create channel for message communication. Channels include video, audio and metadata.

NetConnection is the default communication channel with stream ID 0. Protocol and a few command message, including createStream, use the default communication channel.

The command structure from client to server

The command structure from server to client

RTMP Part 4 - Command Message

This part will describe RTMP command message. Server and client in RTMP communicate and request for RPC using command message. Command messages are encoded using Action Message Format (AMF)

AMF have 2 version AMF 0 and AMF 3

Command Message - AMF 0 have a message type id 20 and AMF 3 have a message type id 17.

A command message consist of command name, transaction id and command object.

These command messages can be used to perform connect, createStream, play, etc... It will also be used to update status for the command.

Data Message - AMF 0 have message type id 18 and AMF 3 have message type id 15

These data message are used to send meta data regarding the information of the stream. The meta data can be the configuration data for video and audio.

Shared Object Message - AMF 0 have kMessageContainerEx=19 and AMF 3 have kMessageContainerEx=16

These shared object message are Flash share objects. A flash share object is a collection of value pair information for synchronization across multiple clients.
The following event type are supported

Audio Message - AMF 0 and AMF 3 message type is 8

This message is used to send audio data to peers

Video Message - AMF 0 and AMF 3 message type is 9

This message is used to send video data to peers. As video message is huge and would delay sending of other type of messages, it should set with low priority

Aggregate Message - AMF 0 and AMF 3 message type is 22

It is a single message that contains many sub-messages

There are many type of commands for exchanging communication between peers. The following are the general rules

1. Sender will send a command with a transaction ID and some parameters
2. Receiver process the command and reply a response with same transaction ID
3. The response string can be _result, _error or a method name

2 class objects are used to send various commands

1. NetConnection - it gives a high level representation of connection between server and client
2. NetStream - it represents the channel (video/audio/data) over which data are sent. Control flow command (play, pause, etc...) are sent via NetStream

RTMP Part 3 - Message Format

This part will describe about RTMP message format.

RTMP message is used for communication between client and server for any type of message. It contains a header and a payload

The message header contains

The payload can contain any information include video and audio data.

Protocol control message are messages that are required by RTMP and its chunk message. It has a message id of 1 to 7. ID 1 and 2 are for RTMP chunk message, ID 3 to 6 are for RTMP and ID 7 is for edge server and origin server

Protocol control message must have

1. Message stream id 0
2. Chunk stream id 2
3. Sent as highest priority
4. Fixed size payload

Below are the 7 types of message

Set Chunk Size (ID 1) is used to notify new chunk size to be used

Abort Message (ID 2) is used to notify receiver to discard any partially received message

Acknowledgement (ID 3) is used tell sender that the receiver is receiving bytes equal to the window size. Window size is the maximum number of bytes received without any acknowledgement message

User Control Message (ID 4) is used to notify peers with user control event

The following user control event type are supported

Window Acknowledgement Size (ID 5) is used to inform peers the window size to use when sending acknowledgement

Set Peer Bandwidth (ID 6) is used to inform output bandwidth for the peers

Thursday, October 28, 2010

RTMP Part 2 - Chunks Message

Part 1, I mention about RTMP handshaking. After a successful handshake, RTMP will send message for streaming. In this part, I will mention more about chunking.

RTMP allows connection to multiples message in chunks. Each chunk has a unique chunk stream id so that receiver can use this id to reassemble the chunk into message.

Chunk size is configurable with maximum 65536 and minimum 128 bytes in size.

In general, chunk has the following format

The Basic Header is 1 to 3 bytes

fmt field is important to identify the chunk message header format to be use. There are 4 types of chunk message header format

Type 0 is 11 bytes long. It must be used for start of the chunk stream.

Type 1 is 7 bytes long. This should be used for message that have variable message size (ie, video data) after Type 0 message is sent.

Type 2 is 3 bytes long. This should be used for message that have fix message size (ie, audio and data) after Type 0 message is sent.

Type 3 has no header. This should be used when message a split into multiple chunks. For example, message with same timestamp after Type 0 and message with same id, size and timestamp.

The field description is as follow. Specifically, you should take note that message stream id is stored in Little Endian format. Also, timestamp is in millisecond.
Extended timestamp is used only when timestamp in chunk message header is set to 0x00FFFFFF. The usage of this field is to handle timestamp that is larger than 0xFFFFFF. If timestamp is larger than 0xFFFFFF, a 4 bytes extended timestamp should be used