what i learned about red5, rtmp and flash

i'm living with red5 for last 5-6 months and i realised i've no entry about red5, i'll try to recover the loss now with one huge entry.

first of all, rtmp's specs are not open, so red5 is a reverse engineering product, and it's a pretty successful project if we consider thah facebook is using it. by the way, adobe had announced they'll open rtmp specs at first half of 2009, but yet they didn't.

rtmp is a protocol on tcp, i don't understand why it isn't udp. tcp is a lossless protocol, so in live video streaming it's hard to prevent delay if bandwith is insufficient. it's not possible to drop packets for incoming stream because you can't touch flash player code. it's only possible to drop packets in server side for stream subscribers, but you've to write some code for this, it's not a configuration thing.

if you need to do some scaling or re-broadcast incoming stream to other red5 servers, you've to be inside the code too. people prefer terracotta[1] for scaling. for re-broadcasting it's impossible to find sample code, but it's possible to find theorical solutions in email lists[2].

by default, red5 uses port 5080 for http, 1935 for rtmp, 8443 for rtmps and 8088 for rtmpt. so company firewalls are problem, generally only allowed port is 80. but we had to serve http for our webservice and rtmp for streaming from the same red5, this means two ports at the same time. we overcomed this obstacle with some servet settings in our web.xml[3]. rtmpt means http actually, transfers stream data with continous http requests. so it's possible to serve your webservice and stream server from the same port by adding this settings to your web.xml:

<servlet>
<servlet-name>rtmpt</servlet>
<servlet-class>org.red5.server.net.rtmpt.RTMPTServlet</servlet>
<load-on-startup>1</load>
</servlet>

<servlet-mapping>
<servlet-name>rtmpt</servlet>
<url-pattern>/fcs/*</url>
</servlet-mapping>

<servlet-mapping>
<servlet-name>rtmpt</servlet>
<url-pattern>/open/*</url>
</servlet-mapping>

<servlet-mapping>
<servlet-name>rtmpt</servlet>
<url-pattern>/close/*</url>
</servlet-mapping>

<servlet-mapping>
<servlet-name>rtmpt</servlet>
<url-pattern>/send/*</url>
</servlet-mapping>

<servlet-mapping>
<servlet-name>rtmpt</servlet>
<url-pattern>/idle/*</url>
</servlet-mapping>

it wasn't easy to make ssl work, we recognized that rtmps does not mean rtmp over ssl, in fact there is no such thing as rtmps, it's more appropriate to say rtmps=rtmpts=https. we were using 0.7.4 early, but it just didn't work with ssl, we moved to 0.8.X versions, after some tries we succeeded to make it work at exact version 0.8-RC2, neither earlier nor later.

and in flash side, you should look flash restrictions before designing anything. for example, sizes of permission boxes in flash is constant in flash, if you create a smaller flash, user can't give permission to access camera/microphone. if user doesn't check remember option in permission box, s/he need to give permission for every flash one by one. if in your screen, a part of the flash is not in eyesight, it's not possible to click on any button on flash or you can't make flash fullscreen without a user click. these are adobe's security precautions. and you have to deal with many security related requirements if you want to serve flash from a different server and your website from another, i advice you to read flash's security policy first. this kind of small things can cause whole design changes totally.

and using shared objects for communication between flash clients and server is not a good idea, red5 implementation has some problems with shared objects. we prefered to use remote function calls (flash.net.NetConnection.call) instead.

we're developing software on linux, so writing actionscript was a question mark for us, but with flex3sdk and ant we did it without problem. however i must warn you, if you want to do some design job there is no mature tool for linux, so it wouldn't be a right choice.

we had to push the lines about quality to see if it's possible to use red5 in closed-circuit broadcasts. we saw that it's possible to catch dvd quality (480 lines) with flash player without much effort. dvd quality requires 7-8 mbit/s bandwidth, by default we couldn't exceed 1 mbit/s limit in upload at first time, no one mentions about this limit in any where. you need to add some code to your application to activate bandwidth controls in order to exceed this limit.

flash player couldn't play incoming stream effectively after 480 lines. flash player is capable to encode camera data with h.263 and vp6 codecs, it can't encode h.264 but can decode it to play. to reduce the amount of data going to server it's important to use h.264. by the way it's possible to re-encode incoming stream in server side with xuggler[4] before broadcasting it. we didn't want to add cpu load in server side so we wanted to encode it in client side. we looked for other alternatives rather than flash player to stream camera data, we couldn't find much, probably because of rtmp's closed specs. there were many rtsp clients, if there were something converting rtmp to rtsp, alternatives would be increased in considerable amounts.

vlc version 0.9 and later have support for playing rtmp but no support for broadcast. meanwhile you need to remove some controls for rtmp connection handshake in vlc code and recompile it in order to make it work with red5. with vlc, playing quality was much better than flash player but with much delay. we couldn't investigate cause of delay yet. for broadcast we found a java code, but we didn't prefer to use it because it was not mature enough. we had to use adobe flash media encoder, that means windows dependency, something that we don't want. by the way you have to remove content type control in red5 codes and recompile it to be able to broadcast over rtmpt in adobe flash media encoder.

quality is more related with video interfaces you use rather than encode-decode issues. to be able to broadcast to red5 server from flash, your computer must recognize your video source as a webcam. if you use a capture card, you can use any source that you can connect to your TV (any TV channel, live screencasts, any camera with composite output, dvr cameras). it's important to move data over digital interfaces such as dvi and hdmi. component interface is the only interface supporting hd quality (1080 lines). but if you want high resolution and digital interfaces on a capture card, prices rises exponentially.

that's what i can remember so far, i can add new information as i remember. i mentioned about some key points casually, if you want some more detail about anything mentioned above, just leave a comment.

[1] http://www.terracotta.org/confluence/display/wiki/Red5+and+Terracotta+POC
[2] http://osflash.org/pipermail/red5_osflash.org/2007-December/017650.html
[3] http://gregoire.org/2009/01/28/rtmpt-and-red5/
[4] http://www.xuggle.com/xuggler/

red5 sample for flowplayer bwchek plugin

to be able to make this rtmp sample work, i had to understand whole flowplayer structure. i want to share the settings i'd done, maybe it could save your time.

index.html:

<html>
<head>
<script type="text/javascript" src="flowplayer-min.js"></script>
</head>
<body>
<div id="info">click below to see most appropriate video to your bandwidth</div>
<a class="rtmp" id="red5"><img src="img/showme.png" /></a>
<script language="JavaScript">
$f("red5", "swf/flowplayer-3.1.0.swf", {
log: {
filter: 'org.flowplayer.bwcheck.*',
level: 'debug'
},

clip: {
url: 'skyandice.flv',
urlResolvers: 'bwcheck',
live: true,
provider: 'rtmp'
},

plugins: {

bwcheck: {
url: 'swf/flowplayer.bwcheck-3.1.0.swf',
netConnectionUrl: 'rtmp://red5ip:1935/bwcheck',
bitrates: [40, 150, 400, 700, 1000],
serverType: 'red5',
rememberBitrate: false,

onBwDone: function(url, chosenBitrate, bitrate) {
var el = document.getElementById("info");
el.innerHTML = "Your speed is: " +bitrate+ "<br />Video file served: " +url;
}
},

rtmp: {
url: 'swf/flowplayer.rtmp-3.1.0.swf',
netConnectionUrl: 'rtmp://red5ip:1935/oflaDemo'
}
}
});
</script>
</body>
</html>

files used:

http://static.flowplayer.org/img/player/btn/showme.png # img altina
http://static.flowplayer.org/video/skyandice-40.flv
http://static.flowplayer.org/video/skyandice-150.flv
http://static.flowplayer.org/video/skyandice-400.flv
http://static.flowplayer.org/video/skyandice-700.flv
http://static.flowplayer.org/video/skyandice-1000.flv
# skyandice*.flv'ler /usr/lib/red5/webapps/oflaDemo/streams altına (artık siz nereye kurduysanız)
http://flowplayer.org/releases/flowplayer.bwcheck/flowplayer.bwcheck-3.1.0.swf
http://flowplayer.org/releases/flowplayer.rtmp/flowplayer.rtmp-3.1.0.swf
http://flowplayer.org/releases/flowplayer/flowplayer-3.1.0.zip
# flowplayer-3.1.0.swf, flowplayer.controls-3.1.0.swf, flowplayer-3.1.0.min.js de bu zip dosyasından geliyor
# tum swfler swf dizini altına

of course you have to install bwcheck and oflaDemo applications at server side from 'http://red5ip:5080/installer' url.

i couldn't find this information when i googled around, i hope it will be useful for other googlers.

to be able to do a job in anarchy

in zeitin we are working in a pretty free environment, it's more close to an anarchy rather than a democracy. our priority list is the only point where anarchy ends. when we were in parkyeri, we had weekly coordination meetings that department heads and co-founders attend to put jobs in order for every department in that week.

we are a small team in zeitin now, so we began to do weekly meetings which everyone attends, and we were trying to solve prioritization problems. but our anarchic system didn't allow us to continue these meetings. so as a solution we preferred to leave this job to one volunteer, giray, probably it was the best solution for a small team.

this week, alper got crazy because of a job i'd done that was not in our priority list. this is a side effect of anarchy, it's not so easy to estimate when and who will react when you do something. afterwards i learned that alper had wanted to do the job i'd done before me, but he had come across with the priority list obstacle when he tried to talk with giray and other guys.

it's more easy to estimate results of your moves in a more hierarchical structure or in a place with some rules, at least you'll know the guys who you're supposed to satisfy. but in an anarchic community you're supposed to satisfy everyone, and this is too ideal, not much possible practically. so there becomes no time without arguments, there is always someone complains about someone or something. we don't manage these complaints in our company, so even if you have nothing to do with a subject, you're finding yourself inside it. in fact, in zeitin there can't be a subject that you're not related with, because if you're working in zeitin, it means you've already accepted that you're ready to take responsibility of any thing related with the company.

i think the freedom we have in zeitin is not offered to any other company employee in the world, including google. people think that parkyeri and zeitin is inspired from google. if it's necessary to compare us with a company, i would prefer to be compared with SUN rather than google. what i can understand from outside, SUN is designed more like a school and employee initiative is really important. google seems like pretending to be some other thing rather than it really is. it says "don't do evil" but it forces android users to have a google account to activate their phones. it uses it's distribution power for monopolization. it makes propoganda by saying that "every employee is free to do what ever s/he wants at %20 of her/his work time", but it doesn't share the small detail that every employee need to take approval of their managers at first. the terms like "manager" is a sign of hierarchy inside. it opens two distinct projects with names chrome and chromium and tries to use community power one sidedly. this kind of things make me suspicious about company's sincerity.

in zeitin we prefer debian because it's the only distribution which could keep freedom as a reason to exist. generally other distributions try to use community power, they have other purposes but their way intersect with free software, so they are pretending to be some other thing than they really are like google does.

i know founders of zeitin, i was there when zeitin was started-up, i can assure you that zeitin is a company where freedom and happiness of employees has the highest priority, i hope it can go on like that.

but i can also assure you that freedom does not mean happiness, it's also not true to show the freedom as a cause of success, but i believe it's possible to be free and successful at the same time.

limitless freedom of everyone means anarchy, in anarchies it's normal to have disagreements that can ruin peace, democracy can be a solution to these problems. but it's not possible to adapt democracy to zeitin, because democracy brings rules with itself, laws means restrictions. in this system, by time, you're becoming subject to some rules you've never heard of.

instead of writing rules, keeping them as community morals in minds can be a solution to this problem, in practice you can allow public sanction to keep rules alive. in this case, there can't be any rule that you didn't hear. "knowing the rule" becomes precondition of a rule's existence.

in last example i experienced, i violated our moral of obeying the priority list, and the first obstacle i came across with was alper. nobody supported me so i was also defeated with public sanction. but i kept going on my way despite every thing, i said "let me do this job or just fire me", and i'd exceeded all the obstacles. most probably i'll get one more punishment with my low performance mark at the end of month, to prevent me doing something like this again. (for today performance evaluation job also is giray's responsibility, he gives marks to every one that affects their salary, so it's his choice to give this punishment or not. performance evaluation is the second point where anarchy breaks.)

the job i wanted to do was changing our releasing methodology, at parkyeri times we were using tarballs for our releases and i was despirately defending the idea that we should use deb packages instead, and we should have a private deb repository to keep our releases in and their dependencies that does not exist in official debian repositories. that's why it was a high-priority job for me but apparently giray was not in the same idea with me. if i'd obeyed giray's priorities we would probably not have a deb repository for the next 6 six years (if we think that parkyeri was a six-year old company which doesn't have a deb repository yet)

it was a critical job, because in an anarchic community, doing something in right time and right place will change too much thing for company's feature, it would trigger birth of a new moral and affect community culture. at first sight it seems like forcing others to your methods, but it's not. because if you force people to something unreasonable, they won't accept it and will resist. of course you must have smart and questioning people around to make it work. it's not enough to determine a methodology also you must stand with it to make it a cultural thing. i'm planning to take responsibility of all release jobs for a while, i'll tell my methodologies to people, i'll get feedback, i'll try to make new procedures mature enough. if i'm on the right way, people will accept them and begin to use them. otherwise i'll stay alone and my new procedures will not live long.

zeitin is a new company, people doesn't have some old habits, so my job will be relatively easy. if i let time pass, it wouldn't be enough to offer a reasonable method, i would also have to fight with old routines.

another way was trying to convince people before starting job. but this way is pretty expensive, it requires doing some meetings. for meetings you'll need to want people to stop what they're doing currently and you'll expect them to be focused on the subject as much as you are. and in this method, generally it becomes different people who offers the idea and who does the job actually. i see this a little risky, because ideas can be so unmature when it's offered, procedures should be kept improved during their life cycle not before.

if you'll do a job in an anarchic community, you should belive what you're doing, you should determine your motivations good, because you shouldn't allow what people says to break your motivation, you should make it a principle to finish what you're started.

as well as not letting people break your motivation, you should also not break other people's motivations. you should be careful about what you're saying, you should keep yourself away from purposeless sentences. but if you have a purpose, you should not avoid saying your word, because if you keep your silence, compony can go to a wrong way that you don't want. if you want some morals to be settled up, you should support them. (though i couldn't obey this rule, i was supporting existence of priority list normally, but i thought breaking rules for this time will have more incomes than the damage it'll create). you should also not avoid your word if you think someone doing something wrong, you should let people fix their mistakes, but always keep the possibility in your mind that you can also be wrong. it's not easy to care your words every time, i'm not so good at it as well, i can't control my patience if i feel someone attacks me, and i can break people's motivation when i become reactive.

actually there becomes many times i lost my peace in zeitin, i prefer peace rather than freedom. but when i look behind, in 3 month time we developed vidi which was definetly would last near 1 year if you try to make it done by a classical firm in same scale with us. and also we are inthe half way of gokada project that we did not officially announced yet, we are currently using thinclient solution of gokada in our company now.

apparently, in anarchy, with some moral rules, it's possible to get progress. as a developer, feel of success, feel of "doing something" is very important. i hope we can succeed to make some sales of our products and keep our company alive.

colorify your logs with vim

we are used to watching logs with "tail -f LOGFILE | ccze -A" command, it was very disturbing for us to look at a log file with any editor, afterwards, without colors. thanks to bekir, he wrote a vim syntax file, and we found peace finally. you can get the syntax file from here, you have to put it in "~/.vim/syntax" folder (if doesn't exist, create), then after opening a log file with vim, if you type ":set syntax=log", your world will be more colorful. if you don't want to say ":set syntax=log" every time, just add "au! BufRead,BufNewFile *log set filetype=log" line to your "~/.vimrc" file. to summarize:

#!/bin/sh
mkdir -p ~/.vim/syntax
cd ~/.vim/syntax
wget --no-check-certificate https://svn.bdgn.net/public/test/bekir/.vim/syntax/log.vim
echo "au! BufRead,BufNewFile *log set filetype=log" >> ~/.vimrc

why should a human being use git?

we started using git instead of subversion at our company a while ago. and i had to defend git against though opposite ideas occasionally, but i can't say i was too successful. so i decided to write my advocacy on paper, maybe i can be more convincing in this way.

actually, "why is git better than other version control systems?" question is answered at here very well.

i can summarize my own motives to prefer, as:

* it allows you to commit some part of your work.
* you will love interactive mode with 'git add -p' command.
* it makes it usual to work on branches.
* it's possible to edit past like merging/splitting previos commits, editing previos commit logs.
* there is no difference between main repository and clonned repository, actually there is no main repository.
* you can make commits with other people's name.
* "git-am" is really making easy to share commits with email.

bekir had told once, during migration from rcs to svn at parkyeri people had reacted as "what now? do the revision of whole repository will change when a file is changed". by migrating to git now, file only commits feature came back, and even more, we are able to do commit by hunk as well.

we can say, there are 3 phases of git commits. first commit is with "git add" to index stage, then second with "git commit" to record changes to branch you work on; and at the end if there is a main repository, it's time for "git push", if there is a main branch, it's time for "git merge".

people may find this unnecessarily complicated, because it was very easy with "svn commit" at old times. first of all, the job done with "svn commit" and in these 3 stages of git is definately not equal. before git, whenever i do a long period development (for ex: one week), i would prefer to use rcs to keep my local history of uncompleted work, and was doing final commit to svn when my work is finished completely. otherwise my patch was becoming too big, and it was becoming impossible to review. using rcs for local history was solving this problem for me, but it was also bringing some new problems as well. git became a perfect medicine for me, i think they had solved local history issue very clean. now during development whenever i got an idea that i'm not so sure with the end, i'm just committing all stuff i want to keep at this point to index stage with "git add", then i'm feeling free to do whatever i want on code without any fear. when i realise i'm not on the right way, i'm just returning back to clean point at index stage with "git co ." command. and if i'm supposed to do a three-day job, at the end of every day i'm reviewing what i've done with "git add -p", committing permanent changes to index. in this way instead of a big review at the end of three days, i can do it day by day and i'm reducing possibility to miss some problem during review.

interactive commit mode with "git add -p" is really making your life easier. it prompts every change hunk by hunk, and asking if you accept it or not. by this way, i'm reviewing every change i've done, and adding permanent ones to index, if i see any thing to be fixed before committing, i'm doing the changes at background, when interactive mode ends i'm repeating this process to commit changes i've done at background until all permanent codes are committed, then i'm shooting rubbish with "git co ." command.

some may ask "why you didn't use use svn branches instead of a mix usage of svn-rcs for local history problem? isn't it more reasonable?". believe me i tried, and i can say using branches in svn is not as practical as mentioned in documentations. first of all, why am i opening a branch? to be able to do things i can't do in trunk, right? and what can't i do in trunk? for example i won't prefer to commit something that is experimental, and i won't prefer to move on with one line commits in order not to mess commit log. and i'm now able to do whatever i want in my branch, very nice, but what about merging your changes into trunk? personally i was taking a big diff between the point i forked my branch and the last point i came with it, and i was applying this patch into trunk. by this way i wasn't carrying garbage codes and one line commits to trunk, but i was losing connection between my branch and trunk. i wasn't using "svn merge" command, in other words i was not really using svn branch. the only difference between opening a branch and working with svn-rcs mix usage was publishing my changes for other people. actually i can't say i was never using "svn merge", i was just using it in one direction, for getting changes from trunk to my branch, and i'm remembering i was never sure if X is included or not when i say "svn merge -rX:Y ..." :). after committing everything i've done in my branch to trunk, synchronizing my branch with trunk's new state was another problem. for this, i was returning back to forking point in my branch, i was merging changes done between this point and head in trunk to my branch. it was really becoming troublesome and i gave up using svn branches in a short time and went on with svn-rcs mix usage.

ok, it seems hard to work with branches on svn, isn't it hard with git? actually not, because all the system is designed on branches for git, that's why all branch operations become regular, and all the stuff you try to stay away in svn turns into reflexes in a short time at git. and also "git rebase" command solves many problems. for example, you forked a branch and went on hacking, meanwhile people keeps developing on base branch, only thing you need to do is "git rebase MAINBRANCH" in order to get the further changes to your new branch. and also, now you can feel free to do whatever you want at your new branch, because in git it's possible to change past, you can join your one line commits into a big commit or you can edit a past commit log or you can transfer a specific change to your branch from any branch and from any time with "git cherry-pick" command.

for svn, to be able to change the past you need a command like svnadmin and some privalages, for git there is no main repository, so when you clone a repository, you're really clonning it, you're able to run any command that you can run on the base repository you clonned from. you don't need to send an email to system admins to change something for you.

and "git am" really rules. we were used to do code reviews with emails at parkyeri. when someone comletes the development s/he would send an email labeled with "[PATCH]" keyword on subject and with patch attached, if someone says it's ok to apply, then patch sender would commit. if someone doesn't like it, patch sender would do the related changes and sends a new email. i think this procedure should be better in this way: developer sends patch, the other developer gets it and applies to his/her own repository, sees it works really, does the changes s/he wants and commits it. in this way we may get rid of situations in which same patch is sent 8 times because of small changes. and also to do code review by overlooking from email post is not a good method. when you apply the patch in your repository, you'll find yourself testing the changes as well. so you'll be more likely to catch problems that you can't by overlooking the patch from email. but in this procedure, for svn there is a serious drawback as losing code ownership because you're not able to do commit with another man's name, but for git you can commit with any user info you want. and especially git makes it very easy to share commits with email by "git am" command. it becomes possible to apply a patch sent with email to your repository and to prepare a patch email from your commits at your local branch with one command.

in conclusion, i think git is a well-designed tool, and solved many personal problems i did have. git provides the feautures that other version control systems does provide and beyond. that's why a person who uses git can use any other version control systems, but it's hard to say vice versa. therefore people who are used to common version control systems finds it strange and unneccessarily complicated. i think even if we accept that it's more complicated, it's not unnecessarily! i had produced much more complicated solutions that git solves in a very clean way. and actually i don't agree that it's complicated, it's just different in some way, and after you got used to it, it's not making you more or less exhausted than other version control systems for daily usage.

i think git is the right choice.