HTML5 solution for lipsyncing

General FOSS discussion
1

Anyone in for pitching ideas? I’ve been checking about how HTML5 can quickly build a drawing app, and I realized how relatively easy it was and how possible it is to make a tool with Pencil-like features for pencil-tests, but also sound capabilities. It is the latter sound scrobbling part that I can’t seem to understand, but then again I was never a programmer. To top it all, HTML5 is as easy (or more so) to program with as ActionScript3, though even more efficient to utilize especially in cross-platform.

I was wondering if anyone was interested in helping me create a simple tool for Lipsyncing and Storyboarding? Here are the concepts I had in mind:

  1. At first, only two primary tools: Pencil and eraser.
  2. Multiple Layer support.
  3. Adjustable Framerate, and Pencil/ToonBoom-like exposure timeline (keyframe drawing remains shown in empty frames, until second keyframe is introduced)
  4. Sound Layer, capable of scrobbling.
  5. Light-table/Onion-Skin: previous frame fades by some amount to help animate easily.
  6. Export as PNG (single frame or sequence) and / or LST-file for Synfig.
  7. Press Left and Right to navigate between frames.

Perhaps when this goes well we’ll also consider an X-Sheet export function and other advanced functions, but first let’s start simple and useful. Here’s the interface idea:

1210×843

2

Some random thoughts to move this along. I really like the idea. I think that HTML5 is going to become the way of the future. At some point we won’t be talking about rewriting the Synfig interface in GTK but in HTML5… Anyway, that’s a long way off.

Given that JLipsync is written in java, couldn’t that already be used on a webpage? I tried to do a dumb hack, but there’s obviously a gotcha I’m not aware of.

There are a lot of audio scrubbers that could be used as a simple way to kickstart development. One that looks quite intriguing is bocoup.com/misc/jai/

From reading the spec, though, my concern is that we may not be able to get enough accuracy out of the audio element control. media . currentTime [ = value ] Returns the current playback position, in seconds. Can be set, to seek to the given time. from whatwg.org/specs/web-apps/cu … io-element seems to imply that only an accuracy of a second is possible. Maybe you can set it more accurately than you can read it back.

While I’d like to wish that I have the programming skills to be able to do this, I know that in reality, it’s pretty unlikely…

3

XD HTML5 Synfig is a BRILLIANT idea! Then again, we gotta find someone fluent in the language (it’s actually as easy as Actionscript 3).

Thanks for that Jai reference! I’ll take a look.