[mythtv] Python: collecting searchRecorded into list() has side effects

[Jan]

Am 25.07.2013 15:07, schrieb Raymond Wagner:
> I can't take a closer look currently, but initial guess is that your database connections weren't being properly reused, and you maxed out on allowed connections.
>
>
>
> On Jul 25, 2013, at 7:22, Jan <mythtv at jd67.de> wrote:
>
>> Hi,
>>
>> I try to get a list of Recorded via searchRecorded and print it in given order.
>>
>> As far as I understand searchRecored don't support order by query (ORDER BY).
>>
>> So I collect all Recorded into a list and sort the list ...
>>
>> *** pseudo code **************************************************************
>>
>> # get the
>> rs = list( db.searchRecoreded(recgroup='Default')
>>
>> # do some sorting
>> rs.sort(key=operator.itemgetter('starttime'), reverse=reverse)
>>
>> for r in rs:
>>     print r.title
>>
>> ******************************************************************************
>> at this point everything work as expected.
>>
>> But if I try to get to get the cutpoints from each recorded, if will fail after
>> round about 40 recorded
>>
>> *** more pseudo **************************************************************
>> for i,r in enumerate(rs):
>>     # get cutlist also
>>     print i, r.title, r.markup.getuncutlist()
>>
>> ---
>> 0 Cold Creek Manor []
>> 1 Good Will Hunting []
>> 2 Hairspray []
>> ... snip ...
>> 39 Bad Boys []
>> 40 Bad Boys II---------------------------------------------------------------------------
>> MythDBError                               Traceback (most recent call last)
>> <ipython-input-12-301408f356a5> in <module>()
>>       1 for i,r in enumerate(rs):
>> ----> 2     print i, r.title, r.markup.getuncutlist()
>>       3
>>
>> /usr/lib/python2.7/dist-packages/MythTV/dataheap.pyc in getuncutlist(self)
>>     274             return self._buildlist(self.MARK_CUT_START, self.MARK_CUT_END)
>>     275         def getuncutlist(self):
>> --> 276             return self._buildlist(self.MARK_CUT_END, self.MARK_CUT_START)
>>     277
>>     278     class _Rating( DBDataRef ):
>>
>> /usr/lib/python2.7/dist-packages/MythTV/utility/other.pyc in _buildlist(self, ms, me)
>>     401         start = []
>>     402         stop = []
>> --> 403         for mark in sorted(self, key=lambda m: m.mark):
>>     404             if mark.type == ms:
>>     405                 if len(start) == len(stop):
>>
>> /usr/lib/python2.7/dist-packages/MythTV/database.pyc in __iter__(self)
>>     483
>>     484     def __iter__(self):
>> --> 485         self._populate()
>>     486         return list.__iter__(self)
>>     487
>>
>> /usr/lib/python2.7/dist-packages/MythTV/database.pyc in _populate(self, force, data)
>>     554                 list.append(self, self.SubData(zip(self._datfields, row)))
>>     555         self._populated = True
>> --> 556         self._origdata = self.deepcopy()
>>     557
>>     558     @classmethod
>>
>> /usr/lib/python2.7/dist-packages/MythTV/database.pyc in deepcopy(self)
>>     576     def deepcopy(self):
>>     577         if not self._populated: return []
>> --> 578         return self.fromCopy([dat.copy() for dat in self])
>>     579
>>     580     def revert(self):
>>
>> /usr/lib/python2.7/dist-packages/MythTV/database.pyc in fromCopy(cls, data)
>>     566     @classmethod
>>     567     def fromCopy(cls, data):
>> --> 568         c = cls('', bypass=True)
>>     569         c._populated = True
>>     570         for dat in data: list.append(c, dat)
>>
>> /usr/lib/python2.7/dist-packages/MythTV/database.pyc in __init__(self, where, db, bypass)
>>     526     def __init__(self, where, db=None, bypass=False):
>>     527         list.__init__(self)
>> --> 528         self._db = DBCache(db)
>>     529         self._setClassDefs(self._db)
>>     530         if bypass: return
>>
>> /usr/lib/python2.7/dist-packages/MythTV/database.pyc in __init__(self, db, args, **dbconn)
>>    1261                     break
>>    1262             else:
>> -> 1263                 raise MythDBError(MythError.DB_CREDENTIALS)
>>    1264
>>    1265         # write configuration back into file, ignored if not old format
>>
>> MythDBError: Could not find database login credentials
>>
>> ******************************************************************************
>>
>> If I iterate over searchRecorded (without sorting) everything works be fine:
>>
>> *** more pseudo **************************************************************
>>
>> for r in DB.searchRecorded(recgroup='Default'):
>>     print r.title, r.markup.getuncutlist()
>>
>> ---
>>
>> Cold Creek Manor []
>> ... snip ~ 350 other recorded
>> Tetsuo - The Bullet Man []
>> ******************************************************************************
>>
>> It looks like, if I store the Recorded into a list, a connection will hold
>> for each Recorded and after ~40 Recorded it will run out of connections.
>>
>> QUESTIONS:
>> Is it in general, a bad idea to store many (>40) Recorded objects into a list()?
>>
>> Is there any way to iterate a lot of Recorded in a sorted manner?
>>
>> Help is welcome
>> Jan
>>
>>
>>
I did some debugging and it looks like that the new connections made and 
STORED into DBDataRef._origdata by by deepcopy.
Just for testing I cleared the _origdata and it works perfect.

*** pseudo code 
**************************************************************
for i,r in enumerate( list(DB.searchRecorded(recgroup='Default')) ):
    print i, r.title, r.markup.getuncutlist()
    r.markup._origdata = None # WARNING: THIS IS NOT A FIX OR WORKAROUND
****