[mythtv-commits] [MythTV/mythtv] 75c298: MythTV python: Class System.system: correct typo.

Fri Nov 15 16:53:28 UTC 2019

  Branch: refs/heads/master
  Home:   https://github.com/MythTV/mythtv
  Commit: 75c2982a1e4a8a6631eeae542fceb89d11fe5588
      https://github.com/MythTV/mythtv/commit/75c2982a1e4a8a6631eeae542fceb89d11fe5588
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/system.py

  Log Message:
  -----------
  MythTV python: Class System.system: correct typo.

When calling System.system, the following traceback bails:

Traceback
  File "MythTV/system.py", line 56, in system
    command = command.lsplit(' ',1)
AttributeError: 'str' object has no attribute 'lsplit'

Note the typo: 'lsplit' --> 'split'.
Python has [lr]strip methods, but provides only 'split()' and 'rsplit()'.

  Commit: 0b5b6b9e46d5aa7ecbfa13c1e4e275bf9689606d
      https://github.com/MythTV/mythtv/commit/0b5b6b9e46d5aa7ecbfa13c1e4e275bf9689606d
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/utility/dt.py

  Log Message:
  -----------
  Update MythTV's python binding 'utility/dt.py' according
 patch  from Ticket #13299:
 Handle-timezone-files-with-no-modern-transitions-as-well_fixes_30_master.patch

Note: This handle the corner cases where the zone-info files have no
      'modern transitions' or no transitions at all.

  Commit: 8baf4db3f0bb34b04203a9264c411a848d4b68d2
      https://github.com/MythTV/mythtv/commit/8baf4db3f0bb34b04203a9264c411a848d4b68d2
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/utility/dt.py

  Log Message:
  -----------
  Correct typo in MythTV's python binding datetime.fromRfc().

When calling the method utility.datetime.fromRfc(), the following trace back occurs:
Traceback (most recent call last):
  File "test/test_datetime_001.py", line 232, in test_datetime_001_12
    t_fromrfc = datetime.fromRfc(t)
  File "MythTV/utility/dt.py", line 433, in fromRfc
    return cls(*tz)
TypeError: type object argument after * must be an iterable, not posixtzinfo

Solution: Return the correct datetime object.

  Commit: 6bf6d4db2e283de39bcd3122b8170d6c1df83337
      https://github.com/MythTV/mythtv/commit/6bf6d4db2e283de39bcd3122b8170d6c1df83337
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/altdict.py

  Log Message:
  -----------
  The conversion to/from bool in DictData from altdict.py does not work

as originally assumed: The string '0' or '1' should be converted to
 'True' or 'False', but python interprets 'bool('x')' as follows:

$ python
>>> bool('0')
True
>>> bool('1')
True

We need to convert the string to integer before castin to 'bool':

$ python
>>> bool(int('0'))
False
>>> bool(int('1'))
True
>>>

  Commit: 6333c3bde96972808aa9b8a4d9afb2a75ccf1b4a
      https://github.com/MythTV/mythtv/commit/6333c3bde96972808aa9b8a4d9afb2a75ccf1b4a
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/msearch.py

  Log Message:
  -----------
  Fix logging in Mythtv's python bindings msearch.py:

Logging is defined with those parameters:
MythLog.log(self, mask, level, message, detail=None)

  Commit: 771796578be6b9f127bd00619a334208d512dc36
      https://github.com/MythTV/mythtv/commit/771796578be6b9f127bd00619a334208d512dc36
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/methodheap.py

  Log Message:
  -----------
  Adapt mythpython's MythXML class to new Services/API.

Note: Integers and Bollean values need to be converted to strings.

  Commit: 695e68d78ee3affe47aa2c07bc23fd4e24c32d96
      https://github.com/MythTV/mythtv/commit/695e68d78ee3affe47aa2c07bc23fd4e24c32d96
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/utility/dt.py

  Log Message:
  -----------
  The conversion to timestamps from MythTV's datetime object
 and vice versa does not work if current time is in daylight saving zone
 (dst).

Because of this bug, the methods 'fromEtree' and 'fromJSON' of the class 'Program' of the Python bindings
cannot convert fetched time values of format '2016-03-14T01:59:21Z' to correct timestamps.

For additional info, see
https://stackoverflow.com/questions/8777753/converting-datetime-date-to-utc-timestamp-in-python

I added a revised 'timestamp' method to MythTV's python binding 'datetime' and compared the output:

    def revised_timestamp(self):
         # utc time = local time - utc offset
         utc_naive = self.replace(tzinfo=None) - self.utcoffset()
         utc_epoch = self.utcfromtimestamp(0).replace(tzinfo=None)
         return ((utc_naive - utc_epoch).total_seconds())

    def timestamp(self):
        return time.mktime(self.timetuple()) + self.microsecond/1000000.

In the follwing example, times are taken at
Saturday, June 1, 2019 2:03:48.066 PM GMT+02:00 DST (for variable 'now') and
Saturday, June 1, 2019 2:04:07.954 PM GMT+02:00 DST (for variable('now_utc')

$ python2
>>> from MythTV import datetime
>>> now = datetime.now()
>>> now
datetime(2019, 6, 1, 14, 3, 48, 66210, tzinfo=<MythTV.utility.dt.posixtzinfo object at 0x7ff760301910>)
>>> now.utcoffset()
datetime.timedelta(0, 7200)
>>> now.timestamp()
1559390628.06621

Check https://www.epochconverter.com/ :
Convert epoch to human readable date and vice versa
1559390628.06621
GMT: Saturday, June 1, 2019 12:03:48.066 PM
Your time zone: Saturday, June 1, 2019 2:03:48.066 PM GMT+02:00 DST
Relative: 4 minutes ago

---> That's correct.

>>> now_utc = datetime.utcnow()
>>> now_utc
datetime(2019, 6, 1, 12, 4, 7, 954273, tzinfo=<MythTV.utility.dt.posixtzinfo object at 0x7ff7602f22d0>)
>>> now_utc.utcoffset()
datetime.timedelta(0)
>>> now_utc.timestamp()
1559387047.954273

Check https://www.epochconverter.com/ :
Convert epoch to human readable date and vice versa
1559387047.954273
GMT: Saturday, June 1, 2019 11:04:07.954 AM
Your time zone: Saturday, June 1, 2019 1:04:07.954 PM GMT+02:00 DST

---> That's wrong !

Now let's do the same with the 'revised' method of datetime:
>>> now.revised_timestamp()
1559390628.06621
https://www.epochconverter.com/
Convert epoch to human readable date and vice versa
1559390628.06621
GMT: Saturday, June 1, 2019 12:03:48.066 PM
Your time zone: Saturday, June 1, 2019 2:03:48.066 PM GMT+02:00 DST

---> That's correct.

>>> now_utc.revised_timestamp()
1559390647.954273

https://www.epochconverter.com/
1559390647.954273
GMT: Saturday, June 1, 2019 12:04:07.954 PM
Your time zone: Saturday, June 1, 2019 2:04:07.954 PM GMT+02:00 DST

---> That's correct as well!

  Commit: 5717db1b5e782d8db47c43106ed208a568609df9
      https://github.com/MythTV/mythtv/commit/5717db1b5e782d8db47c43106ed208a568609df9
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/methodheap.py

  Log Message:
  -----------
  Update 'MythBE.getFreeRecorderList' to Myth Protocol 91.

Myth Protocol 87 added GET_FREE_INPUT_INFO, and removed GET_FREE_RECORDER_LIST.
Myth Protocols 89,90,91 changed return values of command GET_FREE_INPUT_INFO.
See definition InputInfo in inputinfo.h.
The Backend Command 'GET_FREE_INPUT_INFO' (Myth Protocol) returns:
A variable length list of InputInfo entries in preferred live TV order.
Returns an empty list if no recorders are available.

This commit changes the method 'getFreeRecorderList' of the class 'MythBE'
to use GET_FREE_INPUT_INFO.
The return type (list of integers) keeps unchanged.

Additionally, it introduces a a method 'getFreeInputInfo' to return
a list of 'InputInfo' tuples of free recorders in preferred live TV order.
The backend command 'GET_FREE_INPUT_INFO 0' returns a list of variable length
containing one or more 'InputInfo' objects.

See definition of InputInfo in inputinfo.h.

InputInfo is a named tuple containing:
('InputInfo', ('name', 'sourceid', 'inputid', 'mplexid', 'chanid', 'displayName',
'recPriority', 'scheduleOrder', 'livetvorder', 'quickTune')).

Usage examples:

MythBE.getFreeInputInfo()[3].displayName   ---> 'Eingang 13:MPEG2TS'

[x.inputid for x in getFreeInputInfo()]   --->  list of free recorders

  Commit: 4572f980406286eafc5447a02c9c0cdee18e0f82
      https://github.com/MythTV/mythtv/commit/4572f980406286eafc5447a02c9c0cdee18e0f82
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/methodheap.py
    M mythtv/bindings/python/MythTV/system.py

  Log Message:
  -----------
  Fix MythSytemEvent class of MythTV's python bindings

MythSytemEvent:

An event is identified by the regex [A-Z0-9_]* .
This includes the system event `KEY_01`.

BEEventConnection:

The class `BEEventConnection` has optional arguements like
`timeout` an `level`. Name then if the arguement order is not followed strictly.

Typo in the call of
"SystemEvent(event['event'], inst.db).command(event)"

Note:
'inst.db' is not a valid class at this level

Uppercase of system event scripts:

According wiki, the default substitution of parameters is given by uppercase,
like '%STARTTIMEUTC%' or %SENDER%'. Stick to that convention.

  Commit: 35ab79122ce5539471ea2544bde73cb143e8a2df
      https://github.com/MythTV/mythtv/commit/35ab79122ce5539471ea2544bde73cb143e8a2df
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/mythproto.py

  Log Message:
  -----------
  Allow storage group paths without trailing slashes in Python
 Binding findfile method.

Use 'os.path.join instead of simply adding strings (path, filename).
This covers both options, paths with or without trailing slashes.

  Commit: 139350393818bc6ea8d767d2ac07d2ba9638949a
      https://github.com/MythTV/mythtv/commit/139350393818bc6ea8d767d2ac07d2ba9638949a
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/mythproto.py

  Log Message:
  -----------
  Various fixes of MythTV Python Bindings in mythproto.py.

Commit eebe25c introduced the new setting 'BackendServerAddr' instead
of 'BackendServerIP[6]'. See ticket #13082.
Fix another occurence of  'BackendServerIP[6]' in class BECache.

In mythtv's protocol #84 the event 'UPDATE_FILE_SIZE' changed to use
'recordedid' instead of tuple ('chanid', 'starttime').
This commit adds an optional parameter 'recordedid' to ftopen(),
but still allows the optional ('chanid', 'starttime') tuple, and
fixes the handling of this event.
See ticket #12365, comment:5.

Fix the optional arguements when openening the control socket in the
class 'FileTransfer'. We need to enable receive events, if we want
listen to them. Long time ago, the commit cd23715 changed this
behaviour of the BEEvent class.

The method 'FileOps.downloadTo()' listen only for events
'DOWNLOAD_FILE UPDATE'. In short file transfers, the event
'DOWNLOAD_FILE FINISHED' is sent without an update event.
Listen for both events when allocating an eventlock.

  Commit: d2d3cb8f76ac4f9072f4a5a53de9046c95044ca2
      https://github.com/MythTV/mythtv/commit/d2d3cb8f76ac4f9072f4a5a53de9046c95044ca2
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/dataheap.py

  Log Message:
  -----------
  Python Bindings Recorded.formatPath() bails if airdate is not
 set.

Calling 'Recorded.formatPath("%U/%T/%pY-%pm-%pd %pH.%pi %T")' on a valid
Recorded instance, gives a traceback.

Recorded.formatPath  without valid airdate:

Traceback (most recent call last):
  File "test/test_Dataheap_Recorded_001.py", line 116, in test_Dataheap_Recorded_001_04
    print(rec.formatPath("%U/%T/%pY-%pm-%pd %pH.%pi %T"))
  File "MythTV/dataheap.py", line 440, in formatPath
    path = path.replace('%o'+tag, airdate.strftime(format))
ValueError: year=1 is before 1900; the datetime strftime() methods require year >= 1900

Setting 'airdate' to
_default_datetime = datetime(1900,1,1, tzinfo=datetime.UTCTZ())
cures it.

  Commit: b97d01bef6fa595ed4c4157b1600aacd71183962
      https://github.com/MythTV/mythtv/commit/b97d01bef6fa595ed4c4157b1600aacd71183962
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/dataheap.py

  Log Message:
  -----------
  The class methods Job.from[Recorded,Program] should return the created class.

Python Bindings: When creating a Job instance with 'Job.fromRecorded' or
Job.fromProgram, the new created instances are not returned, like in the other
'Class.fromSomething' methods of the bindings.

  Commit: 378cfe017a88f1e99d39ae238011145dcea89663
      https://github.com/MythTV/mythtv/commit/378cfe017a88f1e99d39ae238011145dcea89663
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/dataheap.py

  Log Message:
  -----------
  Fix check in Python's Job.fromProgram classmethod.

We want to check the status of a recorded programg, and not the type.
In file static.py, the 'rsRecorded' is a member of RECSTATUS,
therefore check for 'recstatus' instead of 'rectype'.

  Commit: 455cc61805b14ac7cbadb1851eb2818a3d1b1dcd
      https://github.com/MythTV/mythtv/commit/455cc61805b14ac7cbadb1851eb2818a3d1b1dcd
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/utility/other.py

  Log Message:
  -----------
  Logging in the 'deadlinesocket' fails if dealing with utf-8 encoded strings.

The logging of the 'sendheader' and 'recvheader' methods needs unicode,
but the methods itself need to be fed with 'utf-8' encoded data.

See commit e20bd8b for further details.

Since thees methods itself deal with 'utf-8' encoded data, we need to
decode these data to unicode before we send them to the logging instance.

This commit provides and uses a generic function to decode strings to unicode for
python2 and python3, as well.

How to reproduce:

Start python from a PC that has mythtv frontend installed, but it is not currently
running:

When running the MythTV's python binding `BEEventMonitor`
and log events to a file, a traceback occurs:

$ python2 - --nodblog --loglevel debug --verbose all --logfile /tmp/my_logfile
>>> from MythTV import BEEventMonitor
>>> bemon = BEEventMonitor(systemevents=True)
>>> while (True):
....    continue

>From another frontend, start playback of a recording that has a description
with German umlauts or French accents:

Enjoy the traceback:

Unhandled exception in thread started by
     <bound method BEEventConnection.eventloop
         of <MythTV.connections.BEEventConnection object at 0x7f6f922c1ad0>>
Traceback (most recent call last):
  File "MythTV/connections.py", line 427, in eventloop
    self.queueEvents()
  File "MythTV/connections.py", line 396, in queueEvents
    event = self.socket.recvheader(deadline=0.0)
  File "MythTV/utility/other.py", line 387, in recvheader
    'read <-- %d' % size, data)
  File "MythTV/logging.py", line 431, in __call__
    self.log(mask, level, message, detail)
  File "MythTV/logging.py", line 376, in log
    self._logwrite(mask, level, message, detail)
  File "MythTV/logging.py", line 408, in _logfile
    self._LOGFILE.write(buff.getvalue())
  File "/usr/lib/python2.7/codecs.py", line 708, in write
    return self.writer.write(data)
  File "/usr/lib/python2.7/codecs.py", line 369, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 260:
    ordinal not in range(128)

  Commit: 9127e3436b6f6e8487fe355b988a201a7ec3bcfd
      https://github.com/MythTV/mythtv/commit/9127e3436b6f6e8487fe355b988a201a7ec3bcfd
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/_conn_mysqldb.py

  Log Message:
  -----------
  Fix mysql connection of the Python Bindings for python3.

The ticket #11938 shows a bug when connectiong to mysql using the python bindings.

This initial commit shows the initial intention of the author:

[Commit]4b3f512b[/Commit]
or
https://github.com/MythTV/mythtv/commit/980ae04

Let's stick to it.

Note: This will close #11938.

  Commit: e8ab22178c2b6e78871228120fe2a27a2901d2c0
      https://github.com/MythTV/mythtv/commit/e8ab22178c2b6e78871228120fe2a27a2901d2c0
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/connections.py
    M mythtv/bindings/python/MythTV/database.py
    M mythtv/bindings/python/MythTV/dataheap.py
    M mythtv/bindings/python/MythTV/logging.py
    M mythtv/bindings/python/MythTV/methodheap.py
    M mythtv/bindings/python/MythTV/msearch.py
    M mythtv/bindings/python/MythTV/mythproto.py
    M mythtv/bindings/python/MythTV/system.py
    M mythtv/bindings/python/MythTV/utility/__init__.py
    M mythtv/bindings/python/MythTV/utility/dt.py
    M mythtv/bindings/python/MythTV/utility/other.py

  Log Message:
  -----------
  Fix Python Bindings to be compatible to python3 as well.

 - Make deadlinesocket of python bindings compatible to python3, too.

   MythTV's python bindings based on `deadlinesocket` need a
   byte like object in python3.
   Convert everything to/from utf-8 when sending/receiving data from
   the socket.

 - Update iterators and dictionaries of Python Bindings to be compatible
   to python3 as well.

   According porting guides to python3 [1], the methods
   `dict.keys()`, `dict.items()` and `dict.values()` now return
   views instead of lists.
   Unlike lists, a view does not hold copy the data.
   Updates to the underlying dict are reflected in the view.
   Use `list(dict.keys(), dict.items() and dict.values()` as appropriate.

 - Additionally, python2's `next()` method changed to python3 built-in
   function `next()`.

 - Convert `xrange` (python2) to `range` (python3).

 - Fix logging, convert to 'utf-8'.

 - Add a generic function `py23_str` to convert data to unicode/str.

 - Fix python3 handling of time-zone files after applying
   Handle-timezone-files-with-no-modern-transitions-as-well[29/master].patch
   from ticket #13299

  Commit: 834cbb04730ce30c39ef16f53ed9aebb6adc1b8a
      https://github.com/MythTV/mythtv/commit/834cbb04730ce30c39ef16f53ed9aebb6adc1b8a
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/wikiscripts/__init__.py
    M mythtv/bindings/python/MythTV/wikiscripts/wikiscripts.py
    M mythtv/bindings/python/scripts/mythwikiscripts

  Log Message:
  -----------
  Make mythwikiscripts compatible to python3.

Caching files is now different to python2:
Remove /tmp/mythwikiscripts.pickle* prior to testing mythwikiscripts.

  Commit: bca53f0e884f265dbbe0a4954ba6de7199127d9e
      https://github.com/MythTV/mythtv/commit/bca53f0e884f265dbbe0a4954ba6de7199127d9e
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/tmdb3/scripts/populate_locale.py
    M mythtv/bindings/python/tmdb3/scripts/pytmdb3.py
    M mythtv/bindings/python/tmdb3/tmdb3/__init__.py
    M mythtv/bindings/python/tmdb3/tmdb3/cache.py
    M mythtv/bindings/python/tmdb3/tmdb3/cache_engine.py
    M mythtv/bindings/python/tmdb3/tmdb3/cache_file.py
    M mythtv/bindings/python/tmdb3/tmdb3/cache_null.py
    M mythtv/bindings/python/tmdb3/tmdb3/locales.py
    M mythtv/bindings/python/tmdb3/tmdb3/pager.py
    M mythtv/bindings/python/tmdb3/tmdb3/request.py
    M mythtv/bindings/python/tmdb3/tmdb3/tmdb_api.py
    M mythtv/bindings/python/tmdb3/tmdb3/tmdb_auth.py
    M mythtv/bindings/python/tmdb3/tmdb3/util.py
    M mythtv/programs/scripts/metadata/Movie/tmdb3.py

  Log Message:
  -----------
  Add compatibility to python3 to tmdb3 module.

How to check and reproduce:

Check out mythtv from git.

Set the pythonpath to the folder holding the bindings "MythTV":
Example:
export PYTHONPATH=$HOME/dnalor_mythtv/mythtv/mythtv/bindings/python

Create a symlink for tmdb3
$ cd MythTV
$ ln -s ../tmdb3/tmdb3 tmdb3

Copy the folder
mythtv/mythtv/programs/scripts/metadata/Movie
to
$PYTHONPATH/share/mythtv/metadata/Movie

Note: This is noa absolutely necessary, but it is needed for testing
the 'tmdb3' api together with the 'MythTV' bindings.

Clear the tmdb3 cache:
Create a script:
CF=$HOME/.mythtv/cache/pytmdb3.cache
rm $CF
touch $CF

Delete all '*.pyc' files:
find . -name "*.pyc" -type f -delete

Switch to python3:

Make python3 the default when called via `python` or `#!/usr/bin/env python`:
Put a symlink from python3 to python on top of the `PATH` environment:
As root, type
 mkdir -p /opt/python3
 ln -s `which python3` /opt/python3/python

Then in the terminal add the path as first entry:
$ export PATH=/opt/python3/:$PATH

test with
$ env python --version

Run the following tests from the path $PYTHONPATH :

$ python ./share/mythtv/metadata/Movie/tmdb3.py --version
$ python ./share/mythtv/metadata/Movie/tmdb3.py --test
$ python ./share/mythtv/metadata/Movie/tmdb3.py -M -l en 'Indiana Jones'
$ python ./share/mythtv/metadata/Movie/tmdb3.py -M -l de 'Indiana Jones'
$ python ./share/mythtv/metadata/Movie/tmdb3.py -D -l de 217
$ python ./share/mythtv/metadata/Movie/tmdb3.py -l en -C 10
$ python ./share/mythtv/metadata/Movie/tmdb3.py -l en -a US -C 10
$ python ./share/mythtv/metadata/Movie/tmdb3.py -l en -a de -C 10
$ python ./share/mythtv/metadata/Movie/tmdb3.py -l en -a de -C 10 --debug

  Commit: 8ec6a1d1f410ae7aa6a84dbb2665b45ecafc272b
      https://github.com/MythTV/mythtv/commit/8ec6a1d1f410ae7aa6a84dbb2665b45ecafc272b
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/scripts/mythpython

  Log Message:
  -----------
  Make mythpython script compatible to python3.

  Commit: 4eb515926fe8d463d50fb36c16b3cd7412e3c83d
      https://github.com/MythTV/mythtv/commit/4eb515926fe8d463d50fb36c16b3cd7412e3c83d
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/scripts/mythwikiscripts

  Log Message:
  -----------
  Fix missing conversion to python3 in mythwikiscripts.

  Commit: 9be0f39afabc8465e83a1040c1583290e49acadc
      https://github.com/MythTV/mythtv/commit/9be0f39afabc8465e83a1040c1583290e49acadc
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/system.py

  Log Message:
  -----------
  Fix compatibility to python3 in 'VideoGrabber.grabInetref'.

During testing of the use case shown in 'mythvidexport.py',
I found an additional incompatibility to python3.

This patch fixes #12243 as well, by providing the correct
'inetref' id to the grabber script.

  Commit: 6120e8624dc995187a8e5d27800a529d428d9d52
      https://github.com/MythTV/mythtv/commit/6120e8624dc995187a8e5d27800a529d428d9d52
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/tmdb3/scripts/pytmdb3.py
    M mythtv/programs/scripts/metadata/Movie/tmdb3.py

  Log Message:
  -----------
  Fix a leftover in tmdb3 binding and a typo introduced in commit a90e2db

This fixes a typo introduced in one of the previous commits and
corrects python3 compatibiliy in the grabber script tmdb3.py.

  Commit: 7234e888f578fbd45819f75147e06ef70deb8f4b
      https://github.com/MythTV/mythtv/commit/7234e888f578fbd45819f75147e06ef70deb8f4b
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/connections.py
    M mythtv/bindings/python/MythTV/dataheap.py
    M mythtv/bindings/python/MythTV/methodheap.py
    M mythtv/bindings/python/MythTV/mythproto.py

  Log Message:
  -----------
  Python Bindings: Remove deprecation warnings (python3)

Python3 throws deprecation warnings about 'invalid escape sequences'
when runnig from console with the '-W' switch.

Note: Future python 3.x versions expose these warnings to the user and will
treat them as errors, later on.

This commit removes the deprecation warnings, beeing compatible
to python2 and python3

How to check: Run on the devel/python3 branch:

$ python3 -Wd -m compileall -f -q MythTV
    MythTV/connections.py:461: DeprecationWarning: invalid escape sequence \w
      _res_help = {'jump':  re.compile('(\w+)[ ]+- ([\w /,]+)'),
    MythTV/connections.py:463: DeprecationWarning: invalid escape sequence \w
      'query': re.compile('query ([\w ]*\w+)[ \r\n]+- ([\w /,]+)'),
    MythTV/connections.py:464: DeprecationWarning: invalid escape sequence \w
      'play':  re.compile('play ([\w -:]*\w+)[ \r\n]+- ([\w /:,\(\)]+)')}
    MythTV/connections.py:476: DeprecationWarning: invalid escape sequence \.
      reLOC = re.compile('http://(?P<ip>[0-9\.]+):(?P<port>[0-9]+)/.*')
    MythTV/connections.py:604: DeprecationWarning: invalid escape sequence \.
      reLOC = re.compile('http://(?P<ip>[0-9\.]+):(?P<port>[0-9]+)/.*')
    MythTV/dataheap.py:998: DeprecationWarning: invalid escape sequence \s
      sep = '(?:\s?(?:-|/)?\s?)?'
    MythTV/dataheap.py:1002: DeprecationWarning: invalid escape sequence \d
      '(\d{1,4})',
    MythTV/dataheap.py:1004: DeprecationWarning: invalid escape sequence \d
      '(\d{1,3})',
    MythTV/dataheap.py:1007: DeprecationWarning: invalid escape sequence \d
      regex2 = re.compile('(%s(?:Season%s\d*%s)*%s)$' \
    MythTV/methodheap.py:454: DeprecationWarning: invalid escape sequence \[
      bs = BACKEND_SEP.replace('[','\[').replace(']','\]')
    MythTV/methodheap.py:454: DeprecationWarning: invalid escape sequence \]
      bs = BACKEND_SEP.replace('[','\[').replace(']','\]')
    MythTV/methodheap.py:458: DeprecationWarning: invalid escape sequence \.
      '( HOSTNAME (?P<hostname>[a-zA-Z0-9_\.]*))?'
    MythTV/methodheap.py:459: DeprecationWarning: invalid escape sequence \.
      '( SENDER (?P<sender>[a-zA-Z0-9_\.]*))?'
    MythTV/methodheap.py:1146: DeprecationWarning: invalid escape sequence \d
      if re.match('(?:\d{1,3}\.){3}\d{1,3}',backend) or \
    MythTV/mythproto.py:61: DeprecationWarning: invalid escape sequence \d
      _reip = re.compile('(?:\d{1,3}\.){3}\d{1,3}')
    MythTV/mythproto.py:220: DeprecationWarning: invalid escape sequence \[
      'myth://((?P<group>.*)@)?(?P<host>[\[\]a-zA-Z0-9_\-\.]*)(:[0-9]*)?/(?P<file>.*)')
    MythTV/mythproto.py:221: DeprecationWarning: invalid escape sequence \d
      reip = re.compile('(?:\d{1,3}\.){3}\d{1,3}')

  Commit: 623778ee2c18d712abf52d53648937bbeceafb9b
      https://github.com/MythTV/mythtv/commit/623778ee2c18d712abf52d53648937bbeceafb9b
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/programs/scripts/metadata/Movie/tmdb3.py

  Log Message:
  -----------
  Update Movie Grabber tmdb3.py to meet python3 compatibility, again

- Fix another occurence of deprecated next() method

- Check against python2 instead of python3 to be future proof.

Tested with
$ env python3 --version
Python 3.6.8

$ python ....metadata/Movie/tmdb3.py --version
$ python ..../metadata/Movie/tmdb3.py --test
$ python ..../metadata/Movie/tmdb3.py -M -l en 'Indiana Jones'
$ python ..../metadata/Movie/tmdb3.py -M -l de 'Indiana Jones'
$ python ..../metadata/Movie/tmdb3.py -D -l de 217
$ python ..../metadata/Movie/tmdb3.py -l en -C 10
$ python ..../metadata/Movie/tmdb3.py -l en -a US -C 10
$ python ..../metadata/Movie/tmdb3.py -l en -a de -C 10
$ python ..../metadata/Movie/tmdb3.py -l en -a de -C 10 --debug

and python2 as well.

  Commit: 4d9c4ea5ef7d1bd93e15cbdba5ff05c7a15f4857
      https://github.com/MythTV/mythtv/commit/4d9c4ea5ef7d1bd93e15cbdba5ff05c7a15f4857
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/tmdb3/tmdb3/__init__.py
    M mythtv/bindings/python/tmdb3/tmdb3/tmdb_api.py
    M mythtv/bindings/python/tmdb3/tmdb3/util.py

  Log Message:
  -----------
  TMDB3: Add compatibility to python3 for '__repr__' methods

This method is mostly used on terminal sessions like

  >>> from MythTV.tmdb3 import searchMovie
  >>> res = searchMovie('A New Hope')
  >>> res
  <Search Results: A New Hope>
  >>> len(res)
  5
  >>> res[0]
  <Movie 'Star Wars' (1977)>
  >>> res[1]
  <Movie 'New Hope' (2012)>

With python3, one will get an error response like
"TypeError: __repr__ returned non-string (type bytes)"

  Commit: fcc4990946c20135cafdd006787025553e1f2bfc
      https://github.com/MythTV/mythtv/commit/fcc4990946c20135cafdd006787025553e1f2bfc
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/database.py
    M mythtv/bindings/python/MythTV/dataheap.py
    M mythtv/bindings/python/MythTV/mythproto.py
    M mythtv/bindings/python/MythTV/system.py
    M mythtv/bindings/python/MythTV/utility/__init__.py
    M mythtv/bindings/python/MythTV/utility/other.py

  Log Message:
  -----------
  MythTV: Add compatibility to python3 for '__repr__' methods

The 'repr()' method calls the '__repr__' method of an instanciated class in
python and is used mostly in the interactive python shell, like

$ python2
Python 2.7.15+ (default, Nov 27 2018, 23:36:35)

>>> from MythTV import MythDB, Recorded
>>> db = MythDB()
>>> rec = Recorded((3002, 20190622125000), db = db)
>>> rec     # -----> Note: this calls __repr__ under the hood.
b'<Recorded 'Reisezelt - Kurztrip','2019-06-47 12:50:00+02:00' at 0x7fcdb0faabe0>'
>>>

With Python3, this 'repr()' method returns a 'str' type (i.e. unicode),
in contrast to Python2, where this method may return a native 'str' type (ascii)
or an 'utf-8' encoded string.

Using this in a Python3 shell, leads to the following Type-Error:

  $ python3
  Python 3.6.8 (default, Jan 14 2019, 11:02:34)
  >>> from MythTV import MythDB, Recorded
  >>> db = MythDB()
  >>> rec = Recorded((3002, 20190622125000), db = db)
  >>> rec
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: __repr__ returned non-string (type NoneType)
  >>>

This patch adds compatibility to Python3, it simply selects the current
python version when returning data from the '__repr__()' method.

  Commit: d6af02f8e93aaad26e6289a6634a8ab1f033117e
      https://github.com/MythTV/mythtv/commit/d6af02f8e93aaad26e6289a6634a8ab1f033117e
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/utility/dt.py
    M mythtv/bindings/python/MythTV/utility/enum.py

  Log Message:
  -----------
  Add Compatibility to Python3 for metaclasses

Python3 changed the way how to use a metaclass:

See
https://python-future.org/compatible_idioms.html#metaclasses

On python2, one can test it on classes using the 'InputSingleton':

MythTV is using metaclasses for getting the time zone info
in utility/dt.py: The `class posixtzinfo` is designed as
`InputSingleton`, which means that every subsequent call
with the same parameter returns the same object.
With the patches applied from #13299, one gets on python2:

    $ python
    Python 2.7.15+ (default, Nov 27 2018, 23:36:35)
    >>> from MythTV.utility.dt import *
    >>> from MythTV.utility.singleton import InputSingleton

    >>> a = posixtzinfo()
    >>> b=  posixtzinfo('Etc/UTC')
    >>> c = posixtzinfo("America/Anchorage")

    >>> x = posixtzinfo()
    >>> y = posixtzinfo('Etc/UTC')
    >>> z = posixtzinfo("America/Anchorage")

    >>> c == z
    True
    >>> a == x
    True
    >>> b == y
    True

With python3, i get:

    >>> a == x
    False
    >>> b == y
    False
    >>> c == z
    False

which means, the 'InputSingleton' is not applied in the correct way.

Python3 simply ignores the '__metaclass__' property of Python2.

This patch adds the compatibility layer from python-future.

Additionally, this patch fixes an error

  'RuntimeError: dictionary changed size during iteration'

which is needed for python2 to python3 conversion and only occurs if
the '__metaclass__' is not applied correctly (on python3).

  Commit: a2ae8119b446875fe9ea50c32d588f26593d73b6
      https://github.com/MythTV/mythtv/commit/a2ae8119b446875fe9ea50c32d588f26593d73b6
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/connections.py

  Log Message:
  -----------
  Silence warning on exit of python if a mysql cursor is in use

On python3, if you call one of the 'databaseSearch' methods
like in the following sequence 'searchRecorded', you get a
traceback on exit() like this one:

    $ python
    Python 3.6.8 (default, Aug 20 2019, 17:12:48)
    >>> from MythTV import MythDB, Recorded, Program
    >>> db = MythDB()
    >>> reciter = db.searchRecorded(basename = u'3002_201906....mkv')
    >>> rec = next(reciter)
    >>> rec
    <Recorded 'Reisezeit - Kurzflip','2019-06-22 12:50:00+02:00' at..>
    >>> prgm = rec.getRecordedProgram()
    >>> prgm
    <RecordedProgram 'Reisezeit - Kurzflip','2019-06-22 12:54:07.....>
    >>> prgm.title
    'Reisezeit - Kurzflip'
    >>> prgm.videoprop
    'WIDESCREEN,AVC,720'
    >>> exit()
    Exception ignored in: <generator object databaseSearch.__call__ at..>
    Traceback (most recent call last):
      File "..../MythTV/utility/other.py", line 185, in __call__
      File "..../MythTV/_conn_mysqldb.py", line 110, in __exit__
      File "..../MythTV/_conn_mysqldb.py", line 102, in rollback
    _mysql_exceptions.OperationalError:
                                     (2006, 'MySQL server has gone away')

Note: For 'databaseSearch' methods, see
https://www.mythtv.org/wiki/0.25_Python_Bindings/Connection_Handlers
for searchRecorded, searchOldRecorded, searchVideos etc.

Some background info:
MythTV's python bindings use a set of connections to the mysql database,
managed by the class 'DBConnection' and it's parent class
'_Connection_Pool'.
The latter one has a method to delete all aquired connections during
garbage collection ('__del__'), which unconditionally closes all open
connections to the database (opened via the python package MySQLdb,
derived from https://github.com/PyMySQL/mysqlclient-python).

On python2, this '__del__' method is never called on exit, on python3,
this method gets called because of improvements on the garbage collector
(See PEP 442).
Unfortunately, this garbage collection kicks in too early and the opened
cursor claimed by one of the 'databaseSearch' methods gets closed without
finishing the last action ('commit' or 'rollback').
See the tracaback mentioned above.

I verified (see '(*)') that these open connections are closed on exit
by the garbage collection of the package MySQLdb itself, therefore there
is no need to close these connections within the class 'DBConnection' of
the MythTv's bindings explicitely.

(*) Verified by code inspection (in MySQLdb) and observing the
mysql command 'show processlist' as root user in another terminal.

The implemented solution is simple and works for python2 and python3:
Rename the '__del__' method in order to not beeing called by the
pythons garbage colletor of python3.

Final note:
This commit inhibits only a 'warning' on shutdown of a python terminal,
it does not change any behaviour of the python bindings, because the
'databaseSearch' methods only reads from the 'mythconverg' database,
therefore 'commit' or 'roolback' does not do anything on that database.

  Commit: c53c4440229717c0de559f50009b4479b95044f9
      https://github.com/MythTV/mythtv/commit/c53c4440229717c0de559f50009b4479b95044f9
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/methodheap.py

  Log Message:
  -----------
  Allow 'db.seacrchRecorded' method to search by given 'recordedid'

This patch implements a search inside the 'recorded' table by given
'recordedid' by modifing the already implemented 'searchRecorded' method.

Usage:

    $ python
    >>> from MythTV import MythDB
    >>> db = MythDB()
    >>> reciter = db.searchRecorded(recordedid = 4762)
    >>> rec = next(reciter)
    >>> rec
    <Recorded 'The Lady from Shanghai','2019-03-05 13:51:00+01:00' at .>
    >>> exit()

refs #13300

  Commit: 871accb425683b7cd2154d23f9ac77c5142fc47a
      https://github.com/MythTV/mythtv/commit/871accb425683b7cd2154d23f9ac77c5142fc47a
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/bindings/python/MythTV/__init__.py
    M mythtv/bindings/python/MythTV/dataheap.py

  Log Message:
  -----------
  Add the 'recordedfile' table to python bindings and make use of it

This change adds the 'recordedfile' table to the python bindings and
allows the 'Recorded' instance to use (and update) it.

Usage:
    $ python
    >>> from MythTV import MythDB
    >>> db = MythDB()
    >>> reciter = db.searchRecorded(recordedid = 4762)
    >>> rec = next(reciter)
    >>> rec
    <Recorded 'The Lady from Shanghai','2019-03-05 13:51:00+01:00' at ..>
    >>> recfile = rec.getRecordedFile()
    >>> recfile
    <RecordedFile '3030_20190305125100.mkv','4762' at 0x7fb32d108ca8>
    >>> recfile.items()
    [('basename', '3030_20190305125100.mkv'), ('filesize', 2793354377),
    ('width', 992), ('height', 720), ('fps', 25.0), ('aspect', 1.377778),
     ........   )]
    >>> exit()

refs #13300

  Commit: 1df343e9ab7defa284a73390210a65cf2112f17e
      https://github.com/MythTV/mythtv/commit/1df343e9ab7defa284a73390210a65cf2112f17e
  Author: Bill Meek <billmeek at mythtv.org>
  Date:   2019-11-15 (Fri, 15 Nov 2019)

  Changed paths:
    M mythtv/configure
    M mythtv/programs/scripts/hardwareprofile/MultipartPostHandler.py
    M mythtv/programs/scripts/hardwareprofile/config.py
    M mythtv/programs/scripts/hardwareprofile/deleteProfile.py
    M mythtv/programs/scripts/hardwareprofile/devicelist.py
    M mythtv/programs/scripts/hardwareprofile/distros/all.py
    M mythtv/programs/scripts/hardwareprofile/distros/distro.py
    M mythtv/programs/scripts/hardwareprofile/distros/mythtv_data/data_mythtv.py
    M mythtv/programs/scripts/hardwareprofile/distros/mythtv_data/main.py
    M mythtv/programs/scripts/hardwareprofile/distros/mythtv_data/makeopts.py
    M mythtv/programs/scripts/hardwareprofile/distros/mythtv_data/orddict.py
    M mythtv/programs/scripts/hardwareprofile/distros/mythtv_data/request.py
    M mythtv/programs/scripts/hardwareprofile/distros/mythtv_data/uuiddb.py
    M mythtv/programs/scripts/hardwareprofile/fs_util.py
    M mythtv/programs/scripts/hardwareprofile/gate.py
    M mythtv/programs/scripts/hardwareprofile/getLink.py
    M mythtv/programs/scripts/hardwareprofile/hwdata.py
    M mythtv/programs/scripts/hardwareprofile/i18n.py
    M mythtv/programs/scripts/hardwareprofile/os_detect.py
    M mythtv/programs/scripts/hardwareprofile/request.py
    M mythtv/programs/scripts/hardwareprofile/scan.py
    M mythtv/programs/scripts/hardwareprofile/sendProfile.py
    M mythtv/programs/scripts/hardwareprofile/smolt.py
    M mythtv/programs/scripts/hardwareprofile/software.py
    M mythtv/programs/scripts/hardwareprofile/uuiddb.py

  Log Message:
  -----------
  Hardware Profile: Now works python2 and python3

New packages required:
    python-uritools
    python3-uritools
    python-mailer
    python3-mailer
    python-simplejson
    python3-simplejson
    python3-mysqldb # required for bindings too
    python3-future  # required for bindings too

Compare: https://github.com/MythTV/mythtv/compare/952e76461c62...1df343e9ab7d