[mythtv-users] Script to repair recordings with dd

[DaveD]

I have two sources; OTA via an antenna and dual tuner and two cable STBs 
via firewire on a master backend and a slave backend. By looking at the 
recordings with ffprobe, I have determined that the latter (cable) is 
h264 while the former all seems to be mpeg2video and a much higher 
quality and larger file sizes.

I have had this setup for years and, until V.31, all worked well.  Now, 
the cable recordings (which are becoming fewer all the time as I improve 
my antenna system) have trouble.  Occasionally, some attempts at 
playback cause the frontend to fail with "Decoder Error" and, less 
often, even crash the frontend.  Also, mythcommflag hangs at times on 
the h264 recordings and rarely finds commercials accurately.  The OTA 
recordings (mpeg2video) never have trouble with playback or commercial 
flagging.  These problems are discussed in this bug report/discussion: 
https://code.mythtv.org/trac/ticket/13557#comment:13

Unfortunately, I also switched from Nvidia/VDPAU to Intel/VAAPI at the 
same time I upgraded to V.31, so I'm not sure how much of my problems 
are specific to VAAPI, but the commflag problems are unrelated to 
playback, so I suspect it is the highly compressed h264 recordings not 
fitting into the new playback mechanism.

Anyway, the above-mentioned thread gives instructions on how to use dd 
to removed the beginning of the recording, which renders a recording 
that formerly crashed the frontend playable.  I have repaired three 
recordings, so far, and it has worked perfectly each time.  I have 
incorporated the technique into a script and included it, below.  I 
found that ffprobe was giving me durations that were MUCH higher that 
they should be (one 2-hour show reported an 11 hour duration) so, since 
I didn't know what else to look for in ffprobe's output, I used that 
duration and compared it with what should have been, based on a database 
query.  If the duration is more than .2 hours longer than the database 
time, I run dd, strip off the beginning of the file, then rebuild the 
seektable with mythcommflag (per the above-mentioned bug report).

Speaking of ffprobe's output:  the h264 recordings all start with a 
series of error messages:

[h264 @ 0x557c4ec5aa40] SPS unavailable in decode_picture_timing
[h264 @ 0x557c4ec5aa40] non-existing PPS 0 referenced
[h264 @ 0x557c4ec5aa40] SPS unavailable in decode_picture_timing
[h264 @ 0x557c4ec5aa40] non-existing PPS 0 referenced
[h264 @ 0x557c4ec5aa40] decode_slice_header error
[h264 @ 0x557c4ec5aa40] no frame!

This repeats over and over (sent to stderr) anywhere from 20 to 80 (or 
more?) times before it finally gives normal (although sometimes 
inaccurate) clip information.  I'm not sure what to make of it and 
didn't really feel like studying ffprobe to figure it out.

Here's that script.  I run it with cron every AM at 3:30.  I assume 
there will be no more "Decoder Error" messages/crashes.

Dave D.


#!/usr/bin/perl -w

use strict;
use DBI;

my @flist;
my $days = 3;
my $files_checked = 0;
my $bad_files = 0;

# run this script with a filename argument(s) to fix a known problem file(s)
# run without arguments to check all recordings newer than 3 days

if ($#ARGV >= 0) {
     @flist = @ARGV;
}
else {
     @flist = `find /shome/mythtv /e/shome/mythtv -iname "*.ts" 
-daystart -mmin +120 -mtime -$days`;
     chomp @flist;
}

exit if $#flist < 0;

my $dsns = "dbi:mysql:host=localhost;database=mythconverg";
my %attribs = ('RaiseError', '1'); #, 'AutoCommit', '0');
my $dbh;

$dbh = DBI->connect($dsns, 'root', '', \%attribs);

die "database connect error\n" if (!$dbh);

foreach my $f (@flist) {

     my $chan_ID = 0;
     my $fname = '';

     # /e/shome/mythtv/4115_20200705041000.ts
     #print "$f\n";

     # extract the chanid and recording file basename from full path
     if ($f =~ /\/(\d{4})(_\d{14}.ts)$/) {
         $chan_ID = $1;
         $fname = $1 . $2;
         #print "Channel ID is $chan_ID on $f\n";
     }
     else {
         die "Could not get chanid on $f\n";
     }

     # only process recordings from cable box channels, sourceid = 4
     # this query could be eliminated if all recordings are suspect
     my $query = "SELECT sourceid FROM channel WHERE chanid = $chan_ID";
     my $sth = $dbh->prepare($query);
     $sth->execute();
     my $rowref = $sth->fetchrow_arrayref;
     $sth->finish;
     next if $rowref->[0] != 4;

     # select time_to_sec(timediff('2010-09-01 03:00:00', '2010-09-01 
00:10:00' )) / 3600;
     # $start_time was going to indicate if recording was finished, but 
I handled that with -mmin +120 in the find command, above
     $query = "SELECT starttime, time_to_sec(timediff(endtime, 
starttime)) / 3600 FROM recorded WHERE basename = '$fname'";
     $sth = $dbh->prepare($query);
     $sth->execute();
     $rowref = $sth->fetchrow_arrayref;
     my $start_time = $rowref->[0];
     my $db_hours = $rowref->[1];
     $sth->finish;

     $files_checked++;

     my $info = `/usr/bin/ffprobe $f 2>&1 | grep Duration:`;
     chomp $info;

     # "  Duration: 00:29:55.54, start: 11163.435322, bitrate: 3892 kb/s"
     #print "File info for $f is \"$info\"\n";
     if ($info =~ /Duration: (\d\d):(\d\d):(\d\d)\.\d\d/) {
         #print "Duration was $1 hours, $2 minutes\n";
         my $hrs = $1;
         my $mins = $2;
         my $secs = $3;
         my $hours = $hrs + ($mins / 60.0) + ($secs / 3.6e2); # convert 
$hours to floating point and add fraction from $mins

         #printf "%1.3f\n", $hours - $db_hours; # typical file size 
bigger than DB size
         if ($hours > $db_hours + 0.2) { # typical file size bigger than 
DB size by about 0.11 hours
             printf "Database: %1.3f   ffprobe: %1.3f   Diff: %1.3f   ", 
$db_hours, $hours, $hours - $db_hours;
             $bad_files++;
             printf "Bad duration (%s:%s:%s), need to fix %s\n", $hrs, 
$mins, $secs, $f;
             system "/usr/bin/dd bs=1M skip=3 if=$f of=$f.fix";
             system "rm $f";
             system "mv $f.fix $f";
             print "Fixed, running mythcommflag\n";
             system "/usr/bin/mythcommflag -q --rebuild --file $f >> 
/tmp/re-commflag.log 2>&1 &";
         }
         #else {
         #    printf "Database: %1.3f   ffprobe: %1.3f   Diff: %1.3f   
", $db_hours, $hours, $hours - $db_hours;
         #    printf "Valid duration (%s:%s:%s) found in %s\n", $hrs, 
$mins, $secs, $f;
         #}
     }
}

print "Checked duration on $files_checked file";
print 's' if $files_checked != 1;
print ", found $bad_files that need fixing\n";