[mythtv] extremly slow mythfilldatabase in 0.28

[Torbjorn Jansson]

On 2016-07-18 09:10, Karl Dietz wrote:
> On 17.07.2016 23:11, Jonatan Lindblad wrote:
>> Den 2016-07-17 kl. 19:08, skrev Torbjorn Jansson:
>>> On 2016-07-17 13:57, Roger Siddons wrote:
>>
>> <snip>
>>
>>>> The metagrabber shouldn't be run at all by MFD (AFAIK). Add the "-v
>>>> system" log option to confirm it isn't being invoked.
>>>
>>> looks like it is doing just that.
>>
>> The invocation of ttvdb.py seems to come from
>> https://code.mythtv.org/trac/browser/mythtv/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L530.
>>
>>
>>
>> I'm not sure this was taken into account in b64be0f8, but calling the
>> script(s) for potentially every program seems a bit unnecessary. Unclear
>> if this is the cause for long runtime though.
>
> When I wrote that test there was no intention to call the grabber for
> each and every programme.
> My hacking time is limited at the moment, so feel free to adjust it to
> store the configuration variables in global variable or similar.
> An easy first optimization might be to split the condition into two, to
> avoid the expensive test (that presumably calls the grabber now) if the
> attribute does not match the expected string.
>
> https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L519-L539
>
>

attached is a patch that attempts to fix this.
it is lightly tested and seems to do what i wanted.
calls to the grabber scripts is significantly reduced (but not 
eliminated) and mfdb now runs much quicker.

the patch is not the most elegant one especially with how i had to 
"solve" the MetadataDownload Get Movie and tvgrabber calls.
it was not possible to but those two lines in the constructor or a 
segfault would occur, most likely other parts needs to be initialized 
properly first, like the connection to the database.

anyway, "works for me"

-------------- next part --------------
diff --git a/mythtv/programs/mythfilldatabase/filldata.cpp b/mythtv/programs/mythfilldatabase/filldata.cpp
index ec168aa..1191477 100644
--- a/mythtv/programs/mythfilldatabase/filldata.cpp
+++ b/mythtv/programs/mythfilldatabase/filldata.cpp
@@ -311,6 +311,7 @@ bool FillData::GrabDataFromFile(int id, QString &filename)
     ChannelInfoList chanlist;
     QMap<QString, QList<ProgInfo> > proglist;
 
+    xmltv_parser.lateInit();
     if (!xmltv_parser.parseFile(filename, &chanlist, &proglist))
         return false;
 
diff --git a/mythtv/programs/mythfilldatabase/xmltvparser.cpp b/mythtv/programs/mythfilldatabase/xmltvparser.cpp
index 281da9c..68525f3 100644
--- a/mythtv/programs/mythfilldatabase/xmltvparser.cpp
+++ b/mythtv/programs/mythfilldatabase/xmltvparser.cpp
@@ -34,6 +34,12 @@ XMLTVParser::XMLTVParser() : current_year(0)
     current_year = MythDate::current().date().toString("yyyy").toUInt();
 }
 
+void XMLTVParser::lateInit()
+{
+    _movieGrabberPath = MetadataDownload::GetMovieGrabber();
+    _tvGrabberPath = MetadataDownload::GetTelevisionGrabber();
+}
+
 static uint ELFHash(const QByteArray &ba)
 {
     const uchar *k = (const uchar *)ba.data();
@@ -517,7 +523,7 @@ ProgInfo *XMLTVParser::parseProgram(QDomElement &element)
                     }
                 }
                 else if ((info.attribute("system") == "themoviedb.org") &&
-                    (MetadataDownload::GetMovieGrabber().endsWith(QString("/tmdb3.py"))))
+                    (_movieGrabberPath.endsWith(QString("/tmdb3.py"))))
                 {
                     /* text is movie/<inetref> */
                     QString inetrefRaw(getFirstText(info));
@@ -527,7 +533,7 @@ ProgInfo *XMLTVParser::parseProgram(QDomElement &element)
                     }
                 }
                 else if ((info.attribute("system") == "thetvdb.com") &&
-                    (MetadataDownload::GetTelevisionGrabber().endsWith(QString("/ttvdb.py"))))
+                    (_tvGrabberPath.endsWith(QString("/ttvdb.py"))))
                 {
                     /* text is series/<inetref> */
                     QString inetrefRaw(getFirstText(info));
diff --git a/mythtv/programs/mythfilldatabase/xmltvparser.h b/mythtv/programs/mythfilldatabase/xmltvparser.h
index f254e72..054524a 100644
--- a/mythtv/programs/mythfilldatabase/xmltvparser.h
+++ b/mythtv/programs/mythfilldatabase/xmltvparser.h
@@ -17,6 +17,7 @@ class XMLTVParser
 {
   public:
     XMLTVParser();
+    void lateInit();
 
     ChannelInfo *parseChannel(QDomElement &element, QUrl &baseUrl);
     ProgInfo *parseProgram(QDomElement &element);
@@ -25,6 +26,8 @@ class XMLTVParser
 
   private:
     unsigned int current_year;
+    QString _movieGrabberPath;
+    QString _tvGrabberPath;
 };
 
 #endif // _XMLTVPARSER_H_