[mythtv-commits] [MythTV/mythtv] ee9465: TV Grabber: Initial support for TheTVDB v4 API

Fri Sep 24 19:50:47 UTC 2021

  Branch: refs/heads/master
  Home:   https://github.com/MythTV/mythtv
  Commit: ee94654b62dd9db139525ec8afc266042c264404
      https://github.com/MythTV/mythtv/commit/ee94654b62dd9db139525ec8afc266042c264404
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-05-23 (Sun, 23 May 2021)

  Changed paths:
    M mythtv/bindings/python/setup.py
    A mythtv/bindings/python/ttvdbv4/__init__.py
    A mythtv/bindings/python/ttvdbv4/definitions.py
    A mythtv/bindings/python/ttvdbv4/get_api_v4.py
    A mythtv/bindings/python/ttvdbv4/locales.py
    A mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    A mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py
    A mythtv/bindings/python/ttvdbv4/utils.py
    A mythtv/programs/scripts/metadata/Television/ttvdb4.ini
    A mythtv/programs/scripts/metadata/Television/ttvdb4.py
    A mythtv/programs/scripts/metadata/Television/ttvdb4_doctests

  Log Message:
  -----------
  TV Grabber: Initial support for TheTVDB v4 API

The implementation of the TV grabber is based on following documents:
 - https://api.swaggerhub.com/apis/thetvdb/tvdb-api_v_4/
 - https://app.swaggerhub.com/apis-docs/thetvdb/tvdb-api_v_4/4.3.7

It does not use the OpenAPI Generator for the OAS 3 from
https://openapi-generator.tech/
because
 - the announced spec from thetvdb.com is not fully compliant
   to the OAS 3.x Specification at
   https://swagger.io/specification/
 - the implementation for python3 of the OAS 3.x generator is
   experimental and there is some need for improvement

Therefore, I decided to write a parser for the ttvdbv4 spec, the
generated api should be close to the one of the openapi generator and
therefore easy to substitute.

Note: Getting results takes sometimes loner than 3 minutes.

Known problems so far as of API v 4.3.7:

 - OAS 3.0 spec:
   Every {id} value is defined as 'number', which is a float, or at
   leasta decimal value, but the ttvdb api v4 returns and uses it
   as an integer.

 - The 'SearchResult' schema of the spec lists:
   'imageUrl', 'nameTranslated', 'officialList', 'primaryLanguage',
   'primaryType' but the data returned from thetvdb.com are:
   'image_url', 'name_translated', 'official_list', 'primary_language',
   'primary_type'
    Note: camel casing in spec vs. underscore casing in returned
          response

 - The 'SearchResult' schema lists 'nameTranslated' as a string,
   but it is actually a json object defining a list of languages and
   names

 - The 'Season[Base/Extended]Record' defines a 'type' as integer, but
   I got a dictionary according the 'SeasonType' scheme.

 - The 'SeasonExtendedRecord' lists identical episodes multiple times.

 - The 'getSeriesEpisodes' endpoint is a paged query, but the
   'links' section for 'current', 'previous', 'next' pages is broken:
   The 'next' link is always 'None', the other ones have either a
   missing    '/' in their paths, or a double slash '//' in the paths.
   This makes it hard to write a generator for iterating over the
   paged episodes.

 - RemoteIDs: one get:
   'remoteIds': [{'id': 'https://www.warnerbros.com/tv/gilmore-girls',
                  'sourceName': 'Official Website', 'type': 4},
   but the schema 'RemoteID' has only 'id' and 'type', but the 'type
   is never explained in the schema

- 'networks' as part of seriesExtendedRecord should be like
  'networks': [{'abbreviation': 'ITV1',
                'country': 'gbr',
                'id': 328,
                'name': 'ITV1',
                'slug': 'ITV1'}],
   but is now always empty, worked in previous v4.2.x api.
   Note: Only the search option returns a 'network' item.

 - On API 4.3.7:
   The 'meta' option named 'translations' returns an undefined object
   with members:
    - aliases
    - nameTranslations (list of 'Translations')
    - overviewTranslations (list of 'Translations')
   both 'Translations' are redundant ( i.e.: listed twice)

 - seriesExtendedRecord:
   Characters do not show 'PersonName', if 'meta = 'translations' is
   not used in the query.
   Note: 'personName' is the name of the actor itself, not the
   character she/he is playing.
   I would expect, that the character is subject for translations, not
   the real name of the actor.

  Commit: 4fb58c33da24254156b4e008eea02fdd45c063bc
      https://github.com/MythTV/mythtv/commit/4fb58c33da24254156b4e008eea02fdd45c063bc
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-06 (Fri, 06 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/definitions.py
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4_doctests

  Log Message:
  -----------
  Update to ttvdb api 4.3.8

The following items have been fixed in tvdb-api_v_4.3.8:
See
https://api.swaggerhub.com/apis/thetvdb/tvdb-api_v_4/4.3.8

 - The 'SeasonExtendedRecord' lists identical episodes multiple times.
    Fixed in 4.3.8

 - Broken 'links' section in 'getSeriesEpisodes' endpoint
    The 'getSeriesEpisodes' endpoint is a paged query, but the
    'links' section for 'current', 'previous', 'next' pages is broken:
    The 'next' link is always 'None', the other ones have either a
    missing '/' in their paths, or a double slash '//' in the paths.
    This makes it hard to write a generator for iterating over the
    paged episodes.
    Fixed in 4.3.8

 - Missing 'personName' in characters of seriesExtendedRecord
    In the seriesExtendedRecord:
    Characters do not show 'personName', if 'meta = 'translations'
    is not used in the query.
    Note: 'personName' is the name of the actor itself, not the
          character she/he is playing.
    I would expect, that the character is subject for translations,
    not the real name of the actor.
    Fixed in 4.3.8

  Commit: b763179669bd2eb81bd8977a413472083ff0e6c7
      https://github.com/MythTV/mythtv/commit/b763179669bd2eb81bd8977a413472083ff0e6c7
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-06 (Fri, 06 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/definitions.py
    M mythtv/bindings/python/ttvdbv4/get_api_v4.py
    M mythtv/bindings/python/ttvdbv4/locales.py
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py
    M mythtv/bindings/python/ttvdbv4/utils.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4.ini
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4_doctests

  Log Message:
  -----------
  Add purpose to file headers

  Commit: 9d12abb773259e23704525a0200eb86c7e7e1207
      https://github.com/MythTV/mythtv/commit/9d12abb773259e23704525a0200eb86c7e7e1207
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-06 (Fri, 06 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py

  Log Message:
  -----------
  Set unique http User-Agent for thetvdb v4 api

  Commit: 7c79053defa000e4f71214994a8228c211d47826
      https://github.com/MythTV/mythtv/commit/7c79053defa000e4f71214994a8228c211d47826
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-07 (Sat, 07 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py

  Log Message:
  -----------
  Fix output of missing translations in debug mode

  Commit: c12a8d13d6f713d0575cc3bbe62ddfdf5f79fa0d
      https://github.com/MythTV/mythtv/commit/c12a8d13d6f713d0575cc3bbe62ddfdf5f79fa0d
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-07 (Sat, 07 Aug 2021)

  Changed paths:
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py

  Log Message:
  -----------
  Fix handling of the 'ttvdbv4.ini' file

at the global and local locations.

  Commit: 679729ce9e132ceae8e20e7a24451c70d51ccaed
      https://github.com/MythTV/mythtv/commit/679729ce9e132ceae8e20e7a24451c70d51ccaed
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-10 (Tue, 10 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py

  Log Message:
  -----------
  Replace hardcoded definitions of 'People', like 'Actors'

with an Enum class derived from the TTVDBv4 API.
Note: For performance reasons, this class is statically defined.

  Commit: a7d2aa1f8f78958a827b13075d58c4856b56060a
      https://github.com/MythTV/mythtv/commit/a7d2aa1f8f78958a827b13075d58c4856b56060a
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-10 (Tue, 10 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py

  Log Message:
  -----------
  Clarify mythtv grabber version of the http 'User-Agent'.

  Commit: 75baa654e6baea42527d476ff7dfbaa8cfb3fa89
      https://github.com/MythTV/mythtv/commit/75baa654e6baea42527d476ff7dfbaa8cfb3fa89
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-19 (Thu, 19 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/definitions.py
    M mythtv/bindings/python/ttvdbv4/get_api_v4.py
    M mythtv/bindings/python/ttvdbv4/locales.py
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py
    M mythtv/bindings/python/ttvdbv4/utils.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py

  Log Message:
  -----------
  Clarify copyright and license

of the MythTV Python Bindings for TheTVDB v4 API.

  Commit: 0713f955d1367112e3770a3814f6b1b8e585e71b
      https://github.com/MythTV/mythtv/commit/0713f955d1367112e3770a3814f6b1b8e585e71b
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-08-19 (Thu, 19 Aug 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/definitions.py
    M mythtv/bindings/python/ttvdbv4/get_api_v4.py
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py

  Log Message:
  -----------
  Update bindings for TheTVDB v4 API specification

to version 4.3.17.

The openapai specification for TheTVDB v4 API is now on github:
https://github.com/thetvdb/v4-api/blob/main/docs/swagger.yml

Most of the issues referenced in previous commits for these bindings
are solved or addressed in the their github issue tracker.

  Commit: 14e3d7d5c53f06d02501a31a710c661311c236f1
      https://github.com/MythTV/mythtv/commit/14e3d7d5c53f06d02501a31a710c661311c236f1
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-09-19 (Sun, 19 Sep 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/definitions.py
    M mythtv/bindings/python/ttvdbv4/get_api_v4.py
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py

  Log Message:
  -----------
  Update to thetvdb api v4.4.0 (beta)

Things changed in the TTVDB v4 API specification at
https://github.com/thetvdb/v4-api/blob/main/docs/swagger.yml ,
like 'translations', and 'companies' have been added to the
API specification.
Fix re-use of the name-similarity from 'SearchResult' in
'buildCollection'.

  Commit: 5999d40db21e2e7a968596bc3bb9f26e58763c26
      https://github.com/MythTV/mythtv/commit/5999d40db21e2e7a968596bc3bb9f26e58763c26
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-09-19 (Sun, 19 Sep 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py

  Log Message:
  -----------
  Add timestamping in debug output of ttvdb4.py grabber

Count up from start of grabbing in seconds and log the
time difference in seconds.

  Commit: 346c4314c5842f638111a5fff23e2eb1660a63ba
      https://github.com/MythTV/mythtv/commit/346c4314c5842f638111a5fff23e2eb1660a63ba
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-09-20 (Mon, 20 Sep 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4.ini
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py

  Log Message:
  -----------
  Tweak ttvdbv4 code to fit into MythTV's 3 minutes limit

 - The 'search' for title sometimes returns the correct title including
   the year, like `The Forsyte Saga (2002)`.
   Include these items as valid result.
 - Reduce the items of the 'search' result according calculated
   name similarity
 - Tweak threshold of `name_similarity` to 0.6 instead of 0.5

This reduces the runtime on long lasting calls with multiple languages like
`ttvdb4.py -l de -N "Die Munsters" "Der Liebestrank" --debug` or
`ttvdb4.py -l de -N "Gilmore Girls" "Die letzte Klappe"`
significantly.

  Commit: 59cc7e59eede0838f974cd6516fe27a292a1c8cf
      https://github.com/MythTV/mythtv/commit/59cc7e59eede0838f974cd6516fe27a292a1c8cf
  Author: Roland Ernst <rcrernst at gmail.com>
  Date:   2021-09-21 (Tue, 21 Sep 2021)

  Changed paths:
    M mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    M mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py
    M mythtv/programs/scripts/metadata/Television/ttvdb4.ini
    M mythtv/programs/scripts/metadata/Television/ttvdb4.py

  Log Message:
  -----------
  Cache ttvdbv4 bearer token for the 'ttvdb4.py' grabber

for a timespan configured in 'ttvdb4.ini'.

TheTVDB.com says, that this token is valid for 1 month.

Note: In case of a '--debug' switch supplied to 'ttvdb4.py',
the token cache will be updated. This ensures, that we always
use the latest supplied token.

For now, the lifetime is set to 1 day (24 hours). This ensures,
that a mass update of the metadata likely uses the same token.

  Commit: 4e0568a3fdf2b430b75966218f9a11ed8be5b88e
      https://github.com/MythTV/mythtv/commit/4e0568a3fdf2b430b75966218f9a11ed8be5b88e
  Author: rcrdnalor <40721016+rcrdnalor at users.noreply.github.com>
  Date:   2021-09-24 (Fri, 24 Sep 2021)

  Changed paths:
    M mythtv/bindings/python/setup.py
    A mythtv/bindings/python/ttvdbv4/__init__.py
    A mythtv/bindings/python/ttvdbv4/definitions.py
    A mythtv/bindings/python/ttvdbv4/get_api_v4.py
    A mythtv/bindings/python/ttvdbv4/locales.py
    A mythtv/bindings/python/ttvdbv4/myth4ttvdbv4.py
    A mythtv/bindings/python/ttvdbv4/ttvdbv4_api.py
    A mythtv/bindings/python/ttvdbv4/utils.py
    A mythtv/programs/scripts/metadata/Television/ttvdb4.ini
    A mythtv/programs/scripts/metadata/Television/ttvdb4.py
    A mythtv/programs/scripts/metadata/Television/ttvdb4_doctests

  Log Message:
  -----------
  Merge pull request #377 from rcrdnalor/devel/ttvdbv4

Add thetvdb.com API v4 metadata grabber for television

The implementation of the TV grabber is based on following documents:
https://github.com/thetvdb/v4-api/blob/main/docs/swagger.yml (v 4.4.0)

This commit adds an alternative metadata grabber for TV-Series.
The ttvdb4 metadata grabber is up to date with the latest
api definitions of thetvdb.com and follows the intentions in
https://www.mythtv.org/wiki/MythTV_Universal_Metadata_Format

To enable this grabber, run in mythtv frontend
"Setup" -> "Artwork and Data Sources"
and select 'TheTVDatabaseV4' for the default TelevisionGrabber.

Compare: https://github.com/MythTV/mythtv/compare/f605061d607a...4e0568a3fdf2