Veuillez patienter...

Sysord Informatique - 12250 Roquefort-sur-Soulzon

Vos solutions informatiques sur mesure
SOCIETE DE SERVICES EN INGENIERIE INFORMATIQUE

Accueil > Ressources > Whammy

Whammy

Contexte

Nous avons récemment réalisé une application pour une société spécialisée dans le domaine de la télémédecine. Ce logiciel développé en tant qu' application Chrome (hébergée dans le navigateur) nécessitait d'effectuer des captures de vidéos. Maintenant, avec HTML5 il est devenu assez simple d'obtenir le flux vidéo d'une webcam.
En revanche, il est légèrement plus complexe de l'enregistrer, et c'est là qu'intervient Whammy qui permet de sauvegarder le contenu d'un canvas et de l'encoder au format WebM.

Cependant, nous avons rencontré deux problèmes majeurs lors de l'utilisation de Whammy:

Le script original ne permet d'enregistrer que 32 secondes de vidéo;
Whammy génère un Blob qui une fois écrit dans un fichier ne peut plus être lu car l'API FileSystem ne permet pas de lire les fichiers locaux lorsque l'application n'est pas sur un serveur (par contre on peut écrire...).

Cluster

Sans rentrer dans le détail des spécifications du format WebM, celui-ci est constitué d'au moins un cluster contenant les données du média (vidéo + audio + ...) et un timecode. Ce timecode, en millisecondes, est un entier non signé de 2 octets qui peut donc aller jusqu'à 32767. Le script Whammy d'origine ne prévoit l'utilisation que d'un seul cluster, d'où la limitation de la vidéo à 32 secondes.

Pour remédier au problème, nous avons modifié le script pour générer des clusters de 30 secondes, ce qui rend désormais possible l'enregistrement de vidéos aussi longues que la mémoire vive le permet.

Blob ou Array

Dans un soucis de performances nous n'encodons pas les frames lors de la capture mais nous les enregistrons dans un tableau en récupérant l'imageData du canvas. Puis à la fin de la capture les frames sont passées au script Whammy pour qu'elles soient encodées au format webM.

C'est à la fin de cet encodage que Whammy renvoie un Blob de la vidéo que nous ne pouvons pas exploiter dans notre application Chrome.
En effet, l'API FileSystem fournit des fonctions permettant de lire et d'écrire des fichiers sur le système client mais seulement si le script provient d'un serveur.

Pour contourner ce problème nous avons modifié la fonction qui génère le Blob pour qu'elle puisse aussi générer les vidéos sous forme d'un tableau d'octets que l'on enregistre par la suite dans la base de données du navigateur.

Démo

Cette démonstration ne fonctionne que sur les navigateurs supportant l'API getUserMedia. Pour l'instant il y a Chrome, éventuellement une des dernières versions labs d'Opera et la version 16 de Firefox (en beta pour le moment).

Sources

/*
	var vid = new Whammy.Video();
	vid.add(canvas or data url)
	vid.compile()
*/


var Whammy = (function(){
	// in this case, frames has a very specific meaning, which will be 
	// detailed once i finish writing the code

	function toWebM(frames, outputAsArray){
		var info = checkFrames(frames);

		//max duration by cluster in milliseconds
		var CLUSTER_MAX_DURATION = 30000;
		
		var EBML = [
			{
				"id": 0x1a45dfa3, // EBML
				"data": [
					{ 
						"data": 1,
						"id": 0x4286 // EBMLVersion
					},
					{ 
						"data": 1,
						"id": 0x42f7 // EBMLReadVersion
					},
					{ 
						"data": 4,
						"id": 0x42f2 // EBMLMaxIDLength
					},
					{ 
						"data": 8,
						"id": 0x42f3 // EBMLMaxSizeLength
					},
					{ 
						"data": "webm",
						"id": 0x4282 // DocType
					},
					{ 
						"data": 2,
						"id": 0x4287 // DocTypeVersion
					},
					{ 
						"data": 2,
						"id": 0x4285 // DocTypeReadVersion
					}
				]
			},
			{
				"id": 0x18538067, // Segment
				"data": [
					{ 
						"id": 0x1549a966, // Info
						"data": [
							{  
								"data": 1e6, //do things in millisecs (num of nanosecs for duration scale)
								"id": 0x2ad7b1 // TimecodeScale
							},
							{ 
								"data": "whammy",
								"id": 0x4d80 // MuxingApp
							},
							{ 
								"data": "whammy",
								"id": 0x5741 // WritingApp
							},
							{ 
								"data": doubleToString(info.duration),
								"id": 0x4489 // Duration
							}
						]
					},
					{
						"id": 0x1654ae6b, // Tracks
						"data": [
							{
								"id": 0xae, // TrackEntry
								"data": [
									{  
										"data": 1,
										"id": 0xd7 // TrackNumber
									},
									{ 
										"data": 1,
										"id": 0x63c5 // TrackUID
									},
									{ 
										"data": 0,
										"id": 0x9c // FlagLacing
									},
									{ 
										"data": "und",
										"id": 0x22b59c // Language
									},
									{ 
										"data": "V_VP8",
										"id": 0x86 // CodecID
									},
									{ 
										"data": "VP8",
										"id": 0x258688 // CodecName
									},
									{ 
										"data": 1,
										"id": 0x83 // TrackType
									},
									{
										"id": 0xe0,  // Video
										"data": [
											{
												"data": info.width,
												"id": 0xb0 // PixelWidth
											},
											{ 
												"data": info.height,
												"id": 0xba // PixelHeight
											}
										]
									}
								]
							}
						]
					},

					//cluster insertion point
				]
			}
		 ];

						
		//Generate clusters (max duration)
		var frameNumber = 0;
		var clusterTimecode = 0;
		while(frameNumber < frames.length){
			
			var clusterFrames = [];
			var clusterDuration = 0;
			do {
				clusterFrames.push(frames[frameNumber]);
				clusterDuration += frames[frameNumber].duration;
				frameNumber++;				
			}while(frameNumber < frames.length && clusterDuration < CLUSTER_MAX_DURATION);
						
			var clusterCounter = 0;			
			var cluster = {
					"id": 0x1f43b675, // Cluster
					"data": [
						{  
							"data": clusterTimecode,
							"id": 0xe7 // Timecode
						}
					].concat(clusterFrames.map(function(webp){
						var block = makeSimpleBlock({
							discardable: 0,
							frame: webp.data.slice(4),
							invisible: 0,
							keyframe: 1,
							lacing: 0,
							trackNum: 1,
							timecode: Math.round(clusterCounter)
						});
						clusterCounter += webp.duration;
						return {
							data: block,
							id: 0xa3
						};
					}))
				}
			
			//Add cluster to segment
			EBML[1].data.push(cluster);			
			clusterTimecode += clusterDuration;
		}
						
		return generateEBML(EBML, outputAsArray)
	}

	// sums the lengths of all the frames and gets the duration, woo

	function checkFrames(frames){
		var width = frames[0].width, 
			height = frames[0].height, 
			duration = frames[0].duration;
		for(var i = 1; i < frames.length; i++){
			if(frames[i].width != width) throw "Frame " + (i + 1) + " has a different width";
			if(frames[i].height != height) throw "Frame " + (i + 1) + " has a different height";
			if(frames[i].duration < 0 || frames[i].duration > 0x7fff) throw "Frame " + (i + 1) + " has a weird duration (must be between 0 and 32767)";
			duration += frames[i].duration;
		}
		return {
			duration: duration,
			width: width,
			height: height
		};
	}


	function numToBuffer(num){
		var parts = [];
		while(num > 0){
			parts.push(num & 0xff)
			num = num >> 8
		}
		return new Uint8Array(parts.reverse());
	}

	function strToBuffer(str){
		// return new Blob([str]);

		var arr = new Uint8Array(str.length);
		for(var i = 0; i < str.length; i++){
			arr[i] = str.charCodeAt(i)
		}
		return arr;
		// this is slower
		// return new Uint8Array(str.split('').map(function(e){
		// 	return e.charCodeAt(0)
		// }))
	}


	//sorry this is ugly, and sort of hard to understand exactly why this was done
	// at all really, but the reason is that there's some code below that i dont really
	// feel like understanding, and this is easier than using my brain.

	function bitsToBuffer(bits){
		var data = [];
		var pad = (bits.length % 8) ? (new Array(1 + 8 - (bits.length % 8))).join('0') : '';
		bits = pad + bits;
		for(var i = 0; i < bits.length; i+= 8){
			data.push(parseInt(bits.substr(i,8),2))
		}
		return new Uint8Array(data);
	}

	function generateEBML(json, outputAsArray){
		var ebml = [];
		for(var i = 0; i < json.length; i++){
			var data = json[i].data;
			if(typeof data == 'object') data = generateEBML(data, outputAsArray);					
			if(typeof data == 'number') data = bitsToBuffer(data.toString(2));
			if(typeof data == 'string') data = strToBuffer(data);

			if(data.length){
				var z = z;
			}
			
			var len = data.size || data.byteLength || data.length;
			var zeroes = Math.ceil(Math.ceil(Math.log(len)/Math.log(2))/8);
			var size_str = len.toString(2);
			var padded = (new Array((zeroes * 7 + 7 + 1) - size_str.length)).join('0') + size_str;
			var size = (new Array(zeroes)).join('0') + '1' + padded;
			
			//i actually dont quite understand what went on up there, so I'm not really
			//going to fix this, i'm probably just going to write some hacky thing which
			//converts that string into a buffer-esque thing

			ebml.push(numToBuffer(json[i].id));
			ebml.push(bitsToBuffer(size));
			ebml.push(data)
			

		}
		
		//output as blob or byteArray
		if(outputAsArray){
			//convert ebml to an array
			var buffer = toFlatArray(ebml)
			return new Uint8Array(buffer);
		}else{
			return new Blob(ebml, {type: "video/webm"});
		}
	}
	
	function toFlatArray(arr, outBuffer){
		if(outBuffer == null){
			outBuffer = [];
		}
		for(var i = 0; i < arr.length; i++){
			if(typeof arr[i] == 'object'){
				//an array
				toFlatArray(arr[i], outBuffer)
			}else{
				//a simple element
				outBuffer.push(arr[i]);
			}
		}
		return outBuffer;
	}
	
	//woot, a function that's actually written for this project!
	//this parses some json markup and makes it into that binary magic
	//which can then get shoved into the matroska comtainer (peaceably)

	function makeSimpleBlock(data){
		var flags = 0;
		if (data.keyframe) flags |= 128;
		if (data.invisible) flags |= 8;
		if (data.lacing) flags |= (data.lacing << 1);
		if (data.discardable) flags |= 1;
		if (data.trackNum > 127) {
			throw "TrackNumber > 127 not supported";
		}
		var out = [data.trackNum | 0x80, data.timecode >> 8, data.timecode & 0xff, flags].map(function(e){
			return String.fromCharCode(e)
		}).join('') + data.frame;

		return out;
	}

	// here's something else taken verbatim from weppy, awesome rite?

	function parseWebP(riff){
		var VP8 = riff.RIFF[0].WEBP[0];
		
		var frame_start = VP8.indexOf('\x9d\x01\x2a'); //A VP8 keyframe starts with the 0x9d012a header
		for(var i = 0, c = []; i < 4; i++) c[i] = VP8.charCodeAt(frame_start + 3 + i);
		
		var width, horizontal_scale, height, vertical_scale, tmp;
		
		//the code below is literally copied verbatim from the bitstream spec
		tmp = (c[1] << 8) | c[0];
		width = tmp & 0x3FFF;
		horizontal_scale = tmp >> 14;
		tmp = (c[3] << 8) | c[2];
		height = tmp & 0x3FFF;
		vertical_scale = tmp >> 14;
		return {
			width: width,
			height: height,
			data: VP8,
			riff: riff
		}
	}

	// i think i'm going off on a riff by pretending this is some known
	// idiom which i'm making a casual and brilliant pun about, but since
	// i can't find anything on google which conforms to this idiomatic
	// usage, I'm assuming this is just a consequence of some psychotic
	// break which makes me make up puns. well, enough riff-raff (aha a
	// rescue of sorts), this function was ripped wholesale from weppy

	function parseRIFF(string){
		var offset = 0;
		var chunks = {};
		
		while (offset < string.length) {
			var id = string.substr(offset, 4);
			var len = parseInt(string.substr(offset + 4, 4).split('').map(function(i){
				var unpadded = i.charCodeAt(0).toString(2);
				return (new Array(8 - unpadded.length + 1)).join('0') + unpadded
			}).join(''),2);
			var data = string.substr(offset + 4 + 4, len);
			offset += 4 + 4 + len;
			chunks[id] = chunks[id] || [];
			
			if (id == 'RIFF' || id == 'LIST') {
				chunks[id].push(parseRIFF(data));
			} else {
				chunks[id].push(data);
			}
		}
		return chunks;
	}

	// here's a little utility function that acts as a utility for other functions
	// basically, the only purpose is for encoding "Duration", which is encoded as
	// a double (considerably more difficult to encode than an integer)
	function doubleToString(num){
		return [].slice.call(
			new Uint8Array(
				(
					new Float64Array([num]) //create a float64 array
				).buffer) //extract the array buffer
			, 0) // convert the Uint8Array into a regular array
			.map(function(e){ //since it's a regular array, we can now use map
				return String.fromCharCode(e) // encode all the bytes individually
			})
			.reverse() //correct the byte endianness (assume it's little endian for now)
			.join('') // join the bytes in holy matrimony as a string
	}

	function WhammyVideo(speed, quality){ // a more abstract-ish API
		this.frames = [];
		this.duration = 1000 / speed;
		this.quality = quality || 0.8;
	}

	WhammyVideo.prototype.add = function(frame, duration){
		if(typeof duration != 'undefined' && this.duration) throw "you can't pass a duration if the fps is set";
		if(typeof duration == 'undefined' && !this.duration) throw "if you don't have the fps set, you ned to have durations here."
		if('canvas' in frame){ //CanvasRenderingContext2D
			frame = frame.canvas;	
		}
		if('toDataURL' in frame){
			frame = frame.toDataURL('image/webp', this.quality)
		}else if(typeof frame != "string"){
			throw "frame must be a a HTMLCanvasElement, a CanvasRenderingContext2D or a DataURI formatted string"
		}
		if (!(/^data:image\/webp;base64,/ig).test(frame)) {
			throw "Input must be formatted properly as a base64 encoded DataURI of type image/webp";
		}
		this.frames.push({
			image: frame,
			duration: duration || this.duration
		})
	}
	
	WhammyVideo.prototype.compile = function(outputAsArray){
		return new toWebM(this.frames.map(function(frame){
			var webp = parseWebP(parseRIFF(atob(frame.image.slice(23))));
			webp.duration = frame.duration;
			return webp;
		}), outputAsArray)
	}

	return {
		Video: WhammyVideo,
		fromImageArray: function(images, fps, outputAsArray){
			return toWebM(images.map(function(image){
				var webp = parseWebP(parseRIFF(atob(image.slice(23))))
				webp.duration = 1000 / fps;
				return webp;
			}), outputAsArray)
		},
		toWebM: toWebM
		// expose methods of madness
	}
})()

window.URL = window.URL || window.webkitURL;
navigator.getUserMedia  = navigator.getUserMedia || navigator.webkitGetUserMedia ||
                          navigator.mozGetUserMedia || navigator.msGetUserMedia;

window.requestAnimationFrame = (function(){
      return  window.requestAnimationFrame       || 
              window.webkitRequestAnimationFrame || 
              window.mozRequestAnimationFrame    || 
              window.oRequestAnimationFrame      || 
              window.msRequestAnimationFrame
    })();

var video;
var width;
var height;
var canvas;
var images = [];
var ctx;
var result;
var capture;
var loopnum;
var startTime;
var capturing = false;
var msgdiv;
var progress;
var startButton;
var stopButton;

window.onload = function() {

	init();
	var onFailSoHard = function(e) {
	    console.log('Rejected!', e);
	  };

	if (navigator.getUserMedia) {
		navigator.getUserMedia({
			audio : false,
			video : true
		}, function(stream) {
			video.src = window.URL.createObjectURL(stream);
		}, onFailSoHard);
	} else {
	}
};

/**
 * Set the HTML elements we need.
 */
function init() {
	video = document.getElementById('campreview');
	canvas = document.createElement('canvas');
	ctx = canvas.getContext('2d');
	result = document.getElementById('result');
	msgdiv = document.getElementById('information');
	progress = document.getElementById('progress');
	startButton = document.getElementById('startButton');
	stopButton = document.getElementById('stopButton');
}

/**
 * Capture the next frame of the video.
 */
function nextFrame(){
	if(capturing){
		var imageData;
		ctx.drawImage(video, 0, 0, width, height);
		imageData = ctx.getImageData(0, 0, width, height);
		images.push({duration : new Date().getTime() - startTime, datas : imageData});
		startTime = new Date().getTime();
		requestAnimationFrame(nextFrame);
	}else{
		requestAnimationFrame(finalizeVideo);
	}

}

/**
 * Start the encoding of the captured frames.
 */
function finalizeVideo(){
	var capture = new Whammy.Video();
	setMessage('Encoding video...');
	progress.max = images.length;
	showProgress(true);
	encodeVideo(capture, 0);
}

/**
 * Encode the captured frames.
 * 
 * @param capture
 * @param currentImage
 */
function encodeVideo(capture, currentImage) {
	if (currentImage < images.length) {
		ctx.putImageData(images[currentImage].datas, 0, 0);
		capture.add(ctx, images[currentImage].duration);
		delete images[currentImage];
		progress.value = currentImage;
		currentImage++;
		setTimeout(function() {encodeVideo(capture, currentImage);}, 5);
	} else {
		var output = capture.compile();
		result.src = window.URL.createObjectURL(output);
		setMessage('Finished');
		images = [];
		enableStartButton(true);
	}
}

/**
 * Initialize the canvas' size with the video's size.
 */
function initSize() {
	width = video.clientWidth;
	height = video.clientHeight;
	canvas.width = width;
	canvas.height = height;
}

/**
 * Initialize the css style of the buttons and the progress bar
 * when capturing.
 */
function initStyle() {
	setMessage('Capturing...');
	showProgress(false);
	enableStartButton(false);
	enableStopButton(true);
}

/**
 * Start the video capture.
 */
function startCapture() {
	initSize();
	initStyle();
	capturing = true;
	startTime = new Date().getTime();
	nextFrame();
}

/**
 * Stop the video capture.
 */
function stopCapture() {
	capturing = false;
	enableStopButton(false);
}

/* *************************************************************
 *                   Styles functions
 * *************************************************************/

/**
 * Enable/Disable the start button.
 */
function enableStartButton(enabled) {
	startButton.disabled = !enabled;
}

/**
 * Enable/Disable the stop button.
 */
function enableStopButton(enabled) {
	stopButton.disabled = !enabled;
}

/**
 * Display/Hide the progress bar.
 */
function showProgress(show) {
	progress.style.visibility = show ? 'visible' : 'hidden';
}

/**
 * Display a message in the msgdiv block.
 */
function setMessage(message) {
	msgdiv.innerHTML = message;
}

<progress id="progress" style="visibility: hidden;"></progress><br/>

<input class="uiButton" id="startButton" value="Start" type="button" onclick="startCapture();" />

<input class="uiButton" id="stopButton" value="Stop" type="button" onclick="stopCapture();" disabled="disabled" /><br />

<video width="320" height="240" id="campreview" autoplay="true"></video>

<video width="320" height="240" id="result" controls="controls"></video>

Liens

Whammy : https://github.com/antimatter15/whammy
Spécification WebM : http://matroska.org/technical/specs/index.html
Chrome : http://www.google.com/intl/fr/chrome/browser/
Notre Github : https://github.com/sysord/whammy

Whammy

Contexte

Cluster

Blob ou Array

Démo

Sources

Liens

LIENS PARTENAIRES

MENU

SERVICES

CREDITS