| 1.3 | January 25, 2003 | Mention that 720x480 might be possible |
| 1.2.1 | September 8, 2002 | Add ui.itcl for TiVoWeb 1.9.4b3 |
| 1.2 | September 6, 2002 | Recommend vsplit instead of splitstream Recommend mfs_stream instead of mfs_extract Some other tweaks to based on current throughts |
| 1.1.1 | September 4, 2002 | Add warning about TiVoWeb |
| 1.1 | Feb 2, 2002 | Recommend splitstream instead of tyc |
| 1.0 | Jan 10, 2002 | Initial version |
I'm providing personal links for many of the programs listed here, such as tyc and splitstream, because their home pages seem to have become obsolete and I can't find replacements. If you know of the proper URL, let me know so I can update this stuff
This page is an attempt to describe my impressions of Linux (or UNIX or BSD or what-have-you) software which I've tried to use when dealing with TiVo video extraction. Most people are using Windows software, which tends to be a bit more mature and straightforward than Linux multimedia software. When looking for information on doing this stuff under Linux, I was only able to find one other person who had documented their experiences, and his descriptions were somewhat sparse. However, I can't believe that there aren't more people who don't want to use Windows for this, so I figured I'd share my experiences. Note that this isn't supposed to be a complete HOW-TO discussion - I just wanted somewhere to record my thoughts on specific software and give a rundown of my experiences. Also, so much of the "knowledge" of the net these days is contained in web-based bulletin board systems. While these have their advantages, it's somewhat aggravating to continually have to troll through month-old threads to try to find the information you're looking for; I've found that lately the amount of condensed information on the net seems to have been decreasing - everyone simply refers to threads on web bboards, most of which are usually at least 75% crud. Before diving into this, I'd also recommend reading up any informational pages on this subject, even if they're for Windows. Many of the steps are similar and the concepts are always the same. The computer I'm working on is a 1.1GHz Athlon with 512MB of ram running Debian unstable.
This is optional. You can use TiVoWeb to tune your TiVo so that it produces SVCD-mode video right off the bat. This can save you the (time-consuming) re-encoding process you'll otherwise have to go through later. This was discovered by Tictoc in a thread in the DealDatabase Tivo Hacks forum. Install TiVoWeb and go to the Resource editor section. Under Bitrates, choose one of your quality settings to preempt for SVCD - if you've hacked your TiVo to have extra space, you probably want to choose Basic quality since you can afford to record at higher bitrates anyhow. For the tuner input you want to use (for instance, if TiVo is tuning Cable for you, you want to use the entries that start with CATV), set the resolution to "2" and the bitrate to 2500000. Note that you may need to tweak the bitrate to find one that works for your DVD player and allows you to fit enough video on a CD. Obviously, a higher bitrate will allow better video, but at the expense of using more space. Most DVD players apparently begin to fail above about 2.5Mbps; TicToc reports success with up to 2760000. The resolution setting of "2" sets the TiVo to 480x480, one of the official resolutions for SVCD. After changing the resources be sure to hit the "Update Resources" button in TiVoWeb and reboot your TiVo. You probably don't want to fiddle with the "Quality" setting - the TiVo Apparently uses them to differentiate the different recording levels (ie, it doesn't have an impact on image quality). The full list of resolutions is as follows:
| 0 | 352x480 (or maybe 720x480? See below) |
| 1 | 352x480 |
| 2 | 480x480 |
| 3 | 352x240 (?) |
| 4 | 544x480 |
When figuring out what is going to work for you for extraction, it's very handy to have a cross compiler so you can build tools which may be available in source code only form, or to apply patches. The easiest way to do this is simply by installing the cross compiler binaries.
TiVoWeb 1.9.4 still doesn't show program FSIDs by default - these are the code numbers you need to know to extract video from the TiVo. I've made a modified version of ui.itcl for both versions 1.9.3 and 1.9.4 Beta3 available - you can download it, rename it to ui.itcl, replace the one in your tivoweb-tcl/modules directory and restart tivoweb. [ Note: my patch is based on the work Juppers did for TiVoWeb 1.9.2 ] Thereafter in the "NowShowing" list, you should see a list of FSIDs next to each program.
There are a lot of different versions of ExtractStream going around. There is one commonly available binary version, and there are also frequent mentions of the "CVS" ExtractStream and the "TivoApp" ExtractStream. The best one seems to be the one that comes with TivoApp. There is also an ExtractStream derivative called sendstream which may work better on some streams.
However, while all of the ExtractStream variants work differently well for me, the only tool which actually extracts properly is mfs_export from Tridge's TiVo tools. You can either go to the Samba CVS site (http://cvs.samba.org), checkout the "tivo" collection and build it with the cross compiler, or you can get some binaries of mine. There is also an improved version of mfs_export, mfs_stream which will extract multiple FSIDs at once, which is very handy. I usually use this version. Note that while mfs_extract takes its last argument as the output file, this doesn't work with mfs_stream; instead you need to use -s and redirect to a file (example below).
I generally extract using mfs_stream over NFS onto one of
my Linux machines. Be sure to mount with the
rsize=8192,wsize=8192 options - when I let it mount with the
defaults I can only stream about 440kB/sec. Using the 8k
blocks I can get up to about 780kB/sec which is a very
noticeable improvement. So, on the tivo the commands you need
to execute look something like this:
bash-2.02# insmod -f ./nfs-2.0.1.o
This extracts a single tystream, number 239550, to the
NFS-mounted directory. If you have multiple streams you need
to extract, you can automate the process:
bash-2.02# mount -t nfs -o rsize=8192,wsize=8192 10.0.0.1:/tivo /mnt
bash-2.02# ./mfs_stream -s 239550 > /mnt/myvideo.ty
bash-2.02# for fsid in 239550 239560 239563; do ./mfs_stream -s $fsid > /mnt/$fsid.ty; done
Or, to glue them all together which usually makes dealing with
them a lot easier, you can use the fact that mfs_stream can
extract multiple streams at once:
bash-2.02# ./mfs_stream -s 239550 239560 239563 > /mnt/mystream.ty
A tystream is a tivo-specific type of storage format; basically, it's just a wrapper around an MPEG2 stream. To make it usable, the first thing you want to do is convert it from tystream format.
For UNIX users there are a couple of different tools for this. By far the most useful of them is jdiner's vsplit program for decomposing a tystream into MPEG2 audio & video elementary streams. While this is a bit more complicted than how tyc works, it's much more reliable. Unfortunately, jdiner does not currently provide soruce for vsplit so if you're not on x86 you're up the creek. One of the features that vsplit has over its competitors is that it calculates a very accurate time sync value and fixes up some typical problems encountered from extracted streams. If you have a standalone TiVo you should pass the -s switch to vsplit. If you have a DirecTiVo, leave it off. A sample usage would be (Using the mjpegtools mplex):
bash$ vsplit13 -s myvideo.ty myvideo.m2v myvideo.mp2 Processing '3rdman.ty': (10 chunks per tick) Jumping 1 chunks into input (131072 total bytes) Detected Tivo Type: Standalone Detected Audio Stream Type: MPEG Layer II Final standardAudioSize = 880 Final standardFrameLength = 864 Final standardAudioDiff = 3240 or 00:00:00.036 First Video PTS: 00:00:01.203 ......... 100......... 200......... 300......... 400......... 500 deleted .........16100.........16200.........16300........ Found an OOB packet... The Video Diff is: 01:41:45.081 Found an OOB packet... The Audio Diff is: 01:41:45.096 Is it in sequence??? It is OFF by exactly 169586.000000 frames. Nope... Not in sequence... Skipping it... .16400.........16500 deleted A/V Sync Offset: 11ms (i.e. plays 11ms early!) bash$ mplex -f3 -O 11 -o myvideo.mpg myvideo.m2v myvideo.mp2Note that you can ignore the errors about out of sequence packets - this is vsplit fixing up some problems for you. You should pass the Sync Offset that vsplit reports to whatever multiplexor you're using.
Other alternatives include tyc, a
quick Linux hack of the Windows tyConvert program. Usage is
trivial:
bash$ tyc -s < myvideo.ty > myvideo.mpg
It's normal to get one batch of errors right before tyc
finishes. If you get errors throughout the conversion, you've
got a bad tystream. You'll likely need to recruit help if
this happens, make sure you've tried both mfs_stream and the
various permutations of ExtractStream. Assuming you didn't
encounter and bad errors, you now have a useful MPEG stream.
The last is splitstream [source]. Depending on what you're trying to do, splitstream may be more or less useful. It seems to do a better job than tyc, but not as well as vsplit.
You're probably going to want to view progress while
you're working with this stuff. There are several DVD players
available for Linux. I've had the most luck with VLC, the VideoLan Client.
Other popular ones are Xine and OGLE, but they've been more
difficult to get to work out-of-the-box for me so I haven't
bothered with them. Note that while vlc seems to do a pretty
good job of viewing a fully multiplexed MPEG stream, it does
not seem to work on the raw substreams. That is, if you
demultiplex your MPEG stream so you only have the MPEG video
stream, you can't play just that file with vlc - you need to
remultiplex it before vlc will view it. Viewing with vlc is
pretty trivial:
bash$ vlc myvideo.mpg
The one caveat about vlc is that it seems to sometimes play
extracted video slower than it should be played.
There seems to be one graphical MPEG clipper for Linux - GOPchop. Using it, you can clip out commercials to make your video smaller and more seamless. Hint: Turning off "autorefresh" and selectively using the "refresh" button can make it quicker to search for and edit out commercials. Note: While generally I use the MJPEG mpeg2dec, it's much easier to get GOPchop to link against the LiViD mpeg2dec. Unfortunately, GOPchop tends to mangle MPEG files in ways which make the demuxers and other tools very unhappy. Some of its support libraries are also not terribly stable, leading to crashes with some streams.
For glueing clips together and other assorted tasks, I've found mpgtx to be indispensable. If you want to be sure what resolution your mpeg is at, audio details, or whatnot, mpginfo included with it is great. In addition, since you often need to extract a single program as multiple chunks, you can use mpgcat to join the pieces together.
Another set of tools you should have are the Siemens DVB MPEGTools. (These are the dvb-mpegtools package in Debian).
Here is a summary of my experiences with various software, I'd appreciate hearing any hints:
| mjpegtools mpeg2enc | The mjpegtools are your best shot, but they're a bit weird.
First off, you need to make sure that you have the mpeg2dec off of
their page - it has the YUV output switch you'll need to use
mpeg2enc. The general theory of operation is that you use
mpeg2dec to decode to YUV format, and mpeg2enc to encode it back
along a pipeline:
bash$ mpeg2dec -o YUV < video.mpg | mpeg2enc -o new_video.mpg -f 5 -a 2 -b 2200 -I 3 -V 250
The above re-encodes the mpeg stream in video.mpg into SVCD video
at 2.2Mbps. Note that when encoding video, you need to take the
date rate of the audio into account. If you need to crop or scale
the picture, you can insert yuvscaler into the pipeline between
mpeg2dec and mpeg2enc. mpeg2enc has a lot of options, many which
affect the output quality, so the man page is worth a look. Using
the above my output video seems to have bad interlacing artifacts
when viewed on my DVD player. I've found that by using
yuvkineco, which comes with MJPEG tools, I get much better
output. yuvkineco attempts to reverse the interlacing process.
Another thing to note is that while mpeg2enc claims that -f 5 lets
you override the bitrate, I've had a hard time convincing it to
actually do that. Therefore, for film-based recordings, this
seems to have produced the best output, allowing me to get about
45:10 of video onto a single 80 minute CD:
bash$ mpeg2dec -s -o YUV < video.mpg | yuvkineco -F 4 | mpeg2enc -f 3 -o new_video.mpg -B 260 -b 2150 -a 2 -I 0 -n n -V 250
|
| sampeg-2 | Primary drawback is that it can't encode from a standing MPEG file - you need to De-construct the MPEG video into a series of PPM files before it can encode. This is a formidable endeavor, needing about 1GB for every minute of video. dvdview is, however, happy to perform if you want to go down this route. |
| sampeg-4 | Shows a lot of promise, but very flaky. Most recent CVS versions die with an assertion failure for me. Difficult to get to link properly, there seem to be circular dependencies between its support libraries. I also needed to link its support library, libvideogfx, in statically to get it to work, for reasons I can't begin to fathom. In its default form, very slow., averaging about .7-.8fps encoding. Enabling an optimization which supposedly can cause the compiler problems raises this to about 1.6-1.8fps. The video I've gotten it to produce does look fairly good, however. |
| mpeg2_movie | Very easy to use, very fast (About half of real-time on my machine) and seems to generate good video output which the other tools like. However, the quality of the generated video is very sub-par - extremely blocky, to a degree that I wouldn't consider it worthwhile to use |
| fame | Doesn't appear to support MPEG2 |
| m2_tools m2venc | Encodes with lots of errors, output it only a green screen |
| pm2v | Only a library, no front end interface. In any case, the description of the library interface on the web site sounds like it is not flexible enough to do serious encoding |
| LiViD mpeg2enc | Seems too basic to be useful |
You may often find yourself needing to work with separate audio and video streams. For instance, mpeg2enc Only encodes to a raw MPEG2 video file with no Audio. Or perhaps you need to split off the audio to upsample it (see below) and later need to rejoin it.
To generate separate audio/video streams you need a
demultiplex. One possibility I've found is mpgdemux from
mpgtx. Usage is simple:
bash$ mpgdemux -b myvideo myvideo.mpg
This will produce 2 files - myvideo-0.m2v and myvideo-0.mp2.
You can then work on them separately.
Another alternative is bbTOOLS.
Actually a Windows command line program, the author is kind
enough to provide source and it is relatively easy to get it
to compile under Linux. I've made my own "port" of version
1.9
available. Usage is a bit more complex because bbTOOLS
is more flexible; it can extract arbitrary pieces from an
MPEG stream. Generally, you'd be concerned with streams
0xE0 (first video stream) and 0xC0 (first MPEG audio
stream):
bash$ bbdmux my.mpg 0xE0 video.m2v
bash$ bbdmux my.mpg 0xC0 audio.mp2
To join the two files back together again, you need a
multiplexer. I've found both the DVB tools mplex and the
MJPEG tools mplex to be useful.
DVB mplex:
bash$ dvb-mplex -t SVCD -o output.mpg -i ES_STREAM myaudio.mp2 myvideo.m2v
MJPEG mplex:
bash$ mj-mplex -f 5 -o output.mpg -m 2 myaudio.mp2 myvideo.m2v
Depending on what you've done with your streams while they
were split you may find out that, while they're both the
right length, their timing doesn't quite match up right.
You can adjust the timing of the audio and video streams
with the -a and -v options for DVB mplex or the -O option
for MJPEG mplex.
Overall, I've had more luck with the DVB mplexer - its audio/video delay options seem to work better and it seems to do less padding of the output stream or something, producing a smaller output file.
Some DVD players, including my Pioneer DV-440, won't
play MPEG Layer 2 audio at the 32kHz the TiVo records by
default. Unfortunately, to get around this you need to
decompress the audio to a wav file and then re-compress it to
44.1kHz. To do this, you'll need mpg123 (a commonly
available MPEG decoder) to decode the mp2 to a wav file, sox to re-sample it to
44.1kHz, and toolame to
re-encode it. The process is fairly simple. (Use the
demultiplexing described above to obtain the separated mp2
audio file).
bash$ mpg123 --wav myaudio.wav myaudio-0.mp2
It's a shame that audio re-encoding is needed; it's a nasty
process that degrades the quality of the output due to the
lossy MPEG audio encoding. I wouldn't be surprised if the
TiVo could be convinced to output 44.1kHz audio, but it
requires someone to figure out how to do it. You may want
to add the '-r 224' switch to toolame. Although the TiVo
encodes audio at 192kbps, since you're re-encoding it you may
want to give toolame some extra bandwidth headroom to
preserve quality. After you've mucked with your audio you
will of course need to remultiplex it with your video as
described above.
bash$ sox myaudio.wav -r 44100 myaudio-441.wav
bash$ toolame myaudio441.wav myaudio.mp2
The easiest way to create the image to burn is with GNU VCDImager. Usage is
fairly straight forward:
bash$ vcdimager -t svcd -c my.cue -b my.bin foo.mpg
cdrdao burns VCD image files:
bash$ cdrdao write --device 0,0,0 my.cue
Here I'll attempt to summarize the processes I've describe above. There are probably two main possibilities for the way people would want to proceed. (a) You've tuned your TiVo to produce SVCD-quality output right out of the box, you want to extract it, cut out the commercials and burn it. Or, (b) You have video encoded at a setting which is of too high a quality for SVCD, so you need to down-convert it while mucking with it. Unfortunately, I haven't had much luck with (a) yet for reasons I'll go into below. For now, I'll assume everything was happy and explain later in the Caveats section. Obviously, any of the steps described below are optional, depending on your need. For instance, if your input stream has no commercials (Yay, TMC), you wouldn't need to edit them with GOPchop.
In either case, the first thing you need to do is get
the video off of your TiVo. Find the FSIDs and, on your
TiVo, do something like this:
bash# ./mfs_stream -s 10001 10002 10003 > /mnt/mystream.ty
Now, on your PC, use vsplit to convert your tystream
into its MPEG elements and then mux them together to get a
usable MPEG program stream:
bash$ vsplit -s mystream.ty myvideo.m2v myaudio.mp2
...
bash$ mplex -f3 -O 30 -o myvideo.mpg myaudio.mp2 myvideo.m2v
Use GOPchop to edit out commercials and save it as
mychop.mpg:
bash$ GOPchop myvideo.mpg
Note that this step is unfortunately somewhat theoretical -
it's likely to leave you with a botched MPEG which you can't
operate on anymore
Now, if the video is already in SVCD format, skip down
to the step about remuxing your audio. Otherwise, now is
the time to downconvert to SVCD-quality MPEG2. Note that
this will take a LONG TIME.
bash$ mpeg2dec -s -o YUV < mychop.mpg | yuvkineco -F 4 | mpeg2enc -f 3 -o output.mpg -B 260 -b 2150 -a 2 -I 0 -n n -V 250
Depending on how long your video is, you can futz with the
"2150" - that's the bitrate of your video. Go too high and
you either won't be able to fit all your video or you'll
exceed your player's maximum playback speed. Go too low and
the output will look crappy.
If your DVD player is like mine, now is the time to
re-encode your audio at a higher rate so that it will play:
bash$ bbdmux mychop.mpg 0xC0 myaudio.mp2
bash$ mpg123 --wav myaudio.wav myaudio.mp2
bash$ sox myaudio.wav -r 44100 myaudio44.wav
bash$ toolame myaudio44.wav myaudio.mp2
bash$ mplex -f 3 -m 2 -o final.mpg
Finally, image your SVCD and burn it:
bash$ vcdimager --update-scan-offsets -t svcd -c my.cue -b my.bin final.mpg
bash$ cdrdao write --device 0,0,0 my.cue
You should now hopefully have a usable SVCD to play! If you're feeling adventurous, you may want to go explore some of the advanced subjects of authoring VCDs. If you're so inclined, you can add chapters and menus to your VCDs to make them very close to a DVD in terms of functionality.
Unfortunately, so far things haven't gone quite this well for me. The methodology which requires a re-encoding works well for me. However, vcdimager and the various multiplexers are all unhappy with the MPEG streams after they've been edited with GOPchop. This makes it difficult to come up with nicely edited video without doing the quality sacrificing re-encoding step. The re-encoding method sometimes works because mpeg2dec is apparently much more forgiving about its input than many of the other tools - by playing the video through it and re-encoding you basically get rid of the gunk which the other tools don't like. However, progress is being made and hopefully soon there will be a usable approach