Script for multicore CLI render

When the total render time is “big” it is interesting to use the modern multicore CPU in parallel. You know that the better way to export an animation is using a png sequence. Also it allows to split the render into several sessions or (when available) into several cores in parallel.

I start this thread to create a script to perform render in parallel.

If anyone wants to help, you’re welcome. I’m trying to use bash as my first instance. Usage of python, java or perl would be also awesome.

-G

Visual Basic? :stuck_out_tongue:

How about a Makefile:

START=0
END=150 

star.ogg: $(addprefix star.,$(addsuffix .png,$(shell seq $(START) $(END)))) 
        ffmpeg2theora -o $@ -f image2 star.%d.png

star.%.png: star.sifz
        synfig -t png -o $@ --begin-time $*f --end-time $*f $(OPTS) $^

clean:
        rm -f star.*.png star.ogg 

Type “make -j2 star.ogg” to run two synfig processes at once, “make -j4 star.ogg” for 4 processes at once.

It is a shame the synfig -T option doesn’t work as desired.

pixelgeek: yes of course! :stuck_out_tongue:

pabs: thanks! but how does it works? I don’t understand a word of the makefile! :frowning:

star.%.png: star.sifz synfig -t png -o $@ --begin-time $*f --end-time $*f $(OPTS) $^

for the case of png I would like that each processor renders one set of the image sequence. How does that the above code?
:cry:

Does it means that I have to name “star” to the output file? How is divided the number of frames to each processor when I invoke “make” with “-j4” ? (sorry for my ignorance and laziness)

-G

‘star’ is just the example animation I used to perfect it, you can change that to any name.

make understands ‘jobs’ and runs X jobs in different sub-processes, it relies on the operating system (Linux/Darwin/Windows/BSD/etc) to distribute the processes over the available CPU cores. In this Makefile I assign each job to be 1 frame of the animation. The -j specifies the number of jobs to run in parallel.

I tell synfig to render frame X when the filename of the output image is star.X.png.

I tell make that star.ogg depends on all the star.X.png images existing and to run ffmpeg2theora once they do exist.

Here is a simpler version in bash:

#!/bin/bash
name=star
start=0
end=102
frames=$(($end-$start+1))
cores=5
frames_per_core=$(($frames/($cores-1)))
intervals=$(seq 0 $(($cores-1)))

for i in $intervals; do
        b=$(($start+$i*$frames_per_core))
        e=$(($start+($i+1)*$frames_per_core-1))
        test $e -gt $end && e=$end
        echo synfig -t png --begin-time "$b"f --end-time "$e"f $name.sif*
done

wait

ffmpeg2theora -o $name.ogg -f image2 $name.%04d.png

I couldn’t figure out how to set $frames_per_core optimally though, in some cases one core will be left with relatively fewer images to render.

What’s wrong with the -T option in synfig?

I know it only works on unix systems since it actually uses multiple processes (and pipes) instead of multiple threads, but otherwise it should work. Shouldn’t it?

Note: the process forking code is wrong in at least one place, but it luckily doesn’t present itself as a bug.

G.

I don’t see any difference with or without the -T option:

carlos@pcnuevo:~/Descargas$ synfig -b benchmark.sifz synfig(9286) [07:12:01] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg benchmark.sifz ==> benchmark.png: HECHO benchmark.sifz: Rendered in 36.0385 seconds.

carlos@pcnuevo:~/Descargas$ synfig -b -T 4 benchmark.sifz synfig(8277) [14:49:09] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg benchmark.sifz ==> benchmark.png: HECHO benchmark.sifz: Rendered in 37.3492 seconds.

and this is really what I want:

carlos@pcnuevo:~/Descargas$ date && synfig benchmark.sifz --start-time 0f --end-time 5f & synfig benchmark.sifz --start-time 6f --end-time 10f & synfig benchmark.sifz --start-time 11f --end-time 15f && date [1] 9425 [2] 9426 mar abr 21 15:02:25 CEST 2009 synfig(synfig(94269428)) [15:02:25] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg [15:02:25] información : Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(9429) [15:02:25] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg benchmark.sifz ==> benchmark.png: HECHO benchmark.sifz ==> benchmark.png: HECHO benchmark.sifz ==> benchmark.png: HECHO [2]+ Done synfig benchmark.sifz --start-time 6f --end-time 10f mar abr 21 15:02:37 CEST 2009

See how the render time is 12 seconds and I used 3 threads (36/3=12)
-G

I’ve modified the Makefile to only produce png sequence:

[code]START=0
END=15

foo: $(addprefix foo.,$(addsuffix .png,$(shell seq $(START) $(END))))

foo.%.png: benchmark.sifz
synfig -q -t png -o $@ --begin-time $*f --end-time $*f $(OPTS) $^

clean:
rm -f foo.*.png[/code]

and this is the result:

carlos@pcnuevo:~/Descargas$ date && make -j4 foo > /dev/null && date mar abr 21 15:35:21 CEST 2009 synfig(12713) [15:35:21] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12715) [15:35:21] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12714) [15:35:21] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12716) [15:35:21] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12721) [15:35:25] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12722) [15:35:25] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12723) [15:35:25] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12724) [15:35:25] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12733) [15:35:28] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12734) [15:35:29] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12735) [15:35:29] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12736) [15:35:29] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12745) [15:35:32] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12746) [15:35:32] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12747) [15:35:32] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg synfig(12748) [15:35:32] información: Cargando módulos desde /usr/local/etc/synfig_modules.cfg mar abr 21 15:35:36 CEST 2009

15 seconds. Much better but not as quick as send four threads. I guess that there is some waste of time throwing 15 threads instead of 4.

-G

Even with the -T option, synfig only seems to use one CPU on my system at least. It doesn’t fork either according to strace.

Looking at the code, it seems the -T only takes effect when you use quality 0 (the default is 2).

Why would quality relate to the number of threads? Does that make any sense?

Anyway, -Q 0 -T 3 does indeed create (lots of) child processes.

It also shows another problem with that piece of code. Since there is no wait() system call in the synfig code, all the child processes become (and remain) zombies until the main synfig process dies, and the init process finally kills of all them (rather like buffy! :laughing:)

G.

That’s a great idea, although pardon me… the posts don’t make sense.

Are you saying it would be a simple matter to create a .BAT file (if on Windows) by changing the filename of the SYnfig project file within the .BAT? Or is it more complicated than that?

Hi Tushantin.
The idea here is to split the rendering of frames, or sets of frames, across multiple CPUs, rather than having one CPU do everything serially. The problem under Windows is that we can’t easily or programatically target specific CPUs/cores for a specific set of frames to render. So yes, you could use different PCs to render sets of frames, but if you render on one PC under Windows, you’ll end up just using one CPU/core/thread. At which point, you might as well just render as normal.