[mythtv-users] Ubuntu Mythbackend issue

Ashu Desai ashu.desai at gmail.com
Mon May 14 16:06:00 UTC 2018


On Mon, May 14, 2018 at 3:45 AM Stephen Worthington <
stephen_agent at jsw.gen.nz> wrote:

> On Sun, 13 May 2018 23:29:49 -0500, you wrote:
>
> >On Sun, May 13, 2018 at 10:06 PM Stephen Worthington <
> >stephen_agent at jsw.gen.nz> wrote:
> >
> >> On Sun, 13 May 2018 18:56:42 -0500, you wrote:
> >>
> >> >On Sun, May 13, 2018 at 11:55 AM Stephen Worthington <
> >> >stephen_agent at jsw.gen.nz> wrote:
> >> >
> >> >> On Sun, 13 May 2018 10:47:45 -0500, you wrote:
> >> >>
> >> >> >I see "can't write to" issue and some "AutoExpire: CalcParams(): Max
> >> >> >required Free Space: 1.0 GB w/freq: 15 min" thing which when I
> search
> >> >> seems
> >> >> >to be some TV Tuner issue
> >> >>
> >> >> You can ignore the "Autoexpire:" messages - they are normal.  It just
> >> >> means that mythbackend has looked at all the storage groups and
> >> >> checked that they have enough free space.  If they do not, it will
> >> >> choose some recordings to expire to create enough free space.  Always
> >> >> make sure your storage groups have enough free space unless you want
> >> >> old recordings to expire.  I leave a minimum of 20 Gibytes free at
> all
> >> >> times.
> >> >>
> >> >> The "can't write to" messages come about by mythbackend writing a
> zero
> >> >> length file ".test" to each storage group directory on startup.  If
> it
> >> >> fails to create the file, or fails to be able to delete it, it will
> >> >> give the "can't write to" message and will not use that storage group
> >> >> directory for writing until the next time it is started.
> >> >>
> >> >> So, first, take a look and see if your storage group directories
> >> >> contain .test files, and what their ownership and permissions look
> >> >> like.  Then manually delete them if they are there.  If you have had
> a
> >> >> permissions problem that has caused the .test files to be left behind
> >> >> undeleted, then I think the test fails on creating new ones, hence
> the
> >> >> need to manually delete them after you fix the permissions.
> >> >>
> >> >> Then, try creating the .test files yourself as the user that
> >> >> mythbackend runs as (mythtv).  Log in as root and do this:
> >> >>
> >> >> su - mythtv -c touch "<storage group path>/.test"
> >> >>
> >> >> And try deleting them again:
> >> >>
> >> >> su - mythtv -c "rm <storage group path>/.test"
> >> >>
> >> >> If either command does not work, then you need to fix the ownership
> >> >> and permissions until it does work.
> >> >
> >> >
> >> >so i am slightly confused. Here's what I have:
> >> >
> >> >BE user - myth  (no "tv"), mythconverg user: mythtv
> >> >FE user - mythtv
> >> >
> >> >In ubuntu BE, i don't have "root". so i always log in as "myth" and
> when I
> >> >do, it let's me do the touch and rm the .test
> >>
> >> So does the backend user "mythtv" exist and have "mythtv" group?  That
> >> is important.  So what does this command show:
> >>
> >> su - mythtv -c "groups"
> >>
> >> This is what I get:
> >>
> >> root at mypvr:~# su - mythtv -c "groups"
> >> mythtv dialout cdrom audio video
> >>
> >
> >I don't know the password for username "mythtv"
> >During installation, I tried to name my user as "mythtv" for consistency
> >sake, but Ubuntu didn't allow me saying it's reserved so I had to create
> >"myth" as a user
>
> It is quite likely that the "mythtv" user does not have a password. It
> is possible for package installers to create a user without a
> password, and the "mythtv" user is created by the packages.  The
> normal way to access such a user is via su or sudo (or root).  Using a
> different name for your username on the backend box was the correct
> thing to do.  The "myth" user should be a member of the "mythtv"
> group, but it will be a normal user capable of running a desktop and
> running mythtv-setup and mythfrontend.
>
> >I do get mythtv as a username - quick search gave me a command:
> >
> >myth at ScorpioOne:~$ cut -d: -f1 /etc/passwd
> >root
> >daemon
> >bin
> >sys
> >sync
> >games
> >man
> >lp
> >mail
> >news
> >uucp
> >proxy
> >www-data
> >backup
> >list
> >irc
> >gnats
> >nobody
> >systemd-timesync
> >systemd-network
> >systemd-resolve
> >systemd-bus-proxy
> >syslog
> >_apt
> >messagebus
> >uuidd
> >lightdm
> >whoopsie
> >avahi-autoipd
> >avahi
> >dnsmasq
> >colord
> >speech-dispatcher
> >hplip
> >kernoops
> >pulse
> >rtkit
> >saned
> >usbmux
> >myth
> >mysql
> >ntp
> >mythtv
> >sshd
> >vboxadd
> >statd
> >
> >
> >
> >>
> >> I am not sure why it has "dialout", but the other groups are
> >> important.
> >>
> >> >When we test permissions, do we test the "mythtv" because the FE as
> well
> >> as
> >> >the DB user is "mythtv" or do we test "myth" because that's the BE
> user?
> >>
> >> Mythbackend runs as user mythtv on the backend box.  Mythfrontend runs
> >> as whatever user you run your desktop under on the frontend box.  Both
> >> of those users normally have "mythtv" as an additional group on their
> >> respective boxes.  The group IDs of the "mythtv" groups on the two
> >> different boxes will not usually be the same, hence the are not the
> >> same group.
> >>
> >> >I gather the DB user "mythtv" comes in play later once FE talks to the
> BE,
> >> >but the file permissions AND ownership is what confuses me.
> >> >
> >> >My share "/usr/local/media/metadata" is currently showing as
> >> >
> >> >myth at ScorpioOne:~$ ls -ahl /usr/local/media/
> >> >total 208K
> >> >drwxrwsr-x   9 nobody 4294967294   91 May  7 01:07 .
> >> >drwxr-xr-x  11 root   root        127 May 11 14:24 ..
> >> >drwxrwsr-x   4 root   mythtv       35 May  2 08:43 Data
> >> >drwxrwsr-x   2 root   mythtv        6 May  7 01:07 live-tv
> >> >drwxrwsr-x   2 nobody 4294967294 128K May 13  2018 metadata
> >> >drwxrwsr-x 185 nobody 4294967294  20K May 12 23:11 Music
> >> >drwxrwsr-x 103 nobody 4294967294 4.0K Jan  1 09:26 Pics
> >> >drwxrwsr-x   4 root   mythtv       38 May  7 01:34 TV
> >> >drwxrwsr-x   8 root   mythtv      108 Apr 13 12:47 Videos
> >>
> >> For storage groups, the ownership and permissions of the mythbackend
> >> user on the backend box are what matters.  Mythbackend normally runs
> >> as user "mythtv" and group "mythtv" in Ubuntu.  Mythfrontend talks to
> >> mythbackend for access to the storage group files, so for access to
> >> the recordings, its ownership and permissions do not matter as all
> >> that traffic happens over TCP/IP connections between mythbackend and
> >> mythfrontend.  The normal setup for any storage group directory that
> >> mythbackend has to access is to be ownership mythtv:mythtv with user
> >> and group access both set to execute (=directory access), read and
> >> write (chmod u=xrw,g=xrw).
> >>
> >> However, it is possible to access things other than through storage
> >> groups.  Both mythmusic (which is a plugin that runs entirely in
> >> mythfrontend) and mythvideo (which is a builtin part of mythfrontend)
> >> have settings for paths they can use to directly access things via the
> >> normal access paths of the frontend box they are running on.  These
> >> paths are in addition to the storage group settings they also have. So
> >> it depends on how you have them set up - if you have them access
> >> things via their storage groups, then the same rules apply as for any
> >> storage group - access is via mythbackend and the ownership and
> >> permissions of mythbackend are what matter, and access happens from
> >> the backend box.  But if you are using the local system paths, access
> >> to things via those paths uses mythfrontend's ownership and
> >> permissions, and happens from the frontend box.  The local paths are
> >> also different for each frontend - the frontend settings are stored in
> >> the "settings" table with a different hostname field value for each
> >> frontend.  So each frontend can access the videos or music in the
> >> storage groups via the backend, and also potentially different videos
> >> and music stored locally via the frontend.
> >>
> >> I think mythgallery and its replacement (mythimage?) work the same way
> >> as mythmusic and mythvideo.
> >>
> >> Storage directories are set up in mythtv-setup on the backend box. The
> >> local directories for a frontend are set up in mythfrontend on each
> >> frontend box.
> >>
> >> So, I find your /usr/local/media ownership a bit strange.  For a
> >> start, the listing shows that the metadata, Music and Pics directories
> >> have an unknown group (group ID 4294967294).  That normally only
> >> happens if they are network mounted from a different PC, where the
> >> group IDs and user IDs do not match those on the local PC.
> >
> >You are correct - sorry I thought i had mentioned earlier - these are NFS
> >storage.
>
> OK, I must have missed that.
>
> >
> >>   If they
> >> were local they would normally have been created using existing local
> >> users and groups and would be displayed with the name rather than a
> >> number.  Is that what is going on?  That really can cause problems!
> >> And having your recording directories mounted on other boxes means
> >> that the bandwidth to them is limited by the Ethernet connection to
> >> the other boxes, and that can cause failed recordings when the traffic
> >> is too large.
> >>
> >
> >I have a gig ethernet connection to the boxes, both at my home.
> >
> >
> >>
> >> And even more strange is that the /usr/local/media/ directory itself
> >> is nobody:4294967294.  So is that directory a network mount on a
> >> different box?
> >>
> >
> >Yes the storage are NFS mounted for everything
> >
> >   - Video
> >   - TV
> >   - metadata
> >   - Music
> >   - Pics
> >   - live-tv
> >
> >
> >> The other directories (Data, live-tv, TV and Videos) all have root
> >> ownership.  That is not necessarily fatal, as they do have mythtv
> >> group ownership, but it is not normal.  They ought to be mythtv user
> >> ownership.
> >>
> >> And all the files under the storage group directories also should be
> >> mythtv:mythtv ownership.
> >>
> >> It would also be a good idea to check that mythbackend is running as
> >> user mythtv.  Is this something like what "ps -ef | grep mythba" shows
> >> for you?
> >>
> >> root at mypvr:~# ps -ef | grep mythba
> >> mythtv    3480     1  8 May11 ?        06:22:36 /usr/bin/mythbackend
> >> --quiet --syslog local7 -v record,dvbcam
> >> root     18746  7172  0 14:39 pts/2    00:00:00 grep --color=auto
> >> mythba
> >>
> >
> >This is what it shows for me:
> >
> >myth at ScorpioOne:~$ ps -ef | grep mythba
> >myth      4668  2188  0 23:29 pts/18   00:00:00 grep --color=auto mythba
> >mythtv   25292     1  0 May12 ?        00:06:01 /usr/bin/mythbackend
> >--quiet --syslog local7
>
> That is good - mythbackend is running as the mythtv user.
>
>
> I have not used NFS much, but it looks like it has the same problem as
> SAMBA/CIFS - the user IDs and group IDs (apart from root = 0) do not
> match between machines.  That makes it difficult to get ownership to
> work, unless NFS has some mechanism to map the user IDs and group IDs
> to get them to match.  So you probably will need to use wide open
> permissions instead.  As in everybody having access (chmod a=rwx). But
> have a look as the NFS documentation and do a web search to see what
> your options are for getting correct ownership.
>
> Using network drives is fine for read-only things, such as videos,
> pictures and music.  And there should be no problems for the small
> writeable files for things like the metadata and artwork.  But the
> higher bandwidth, and time critical, writing of recordings can be a
> problem.  Looking at the numbers for the bandwidth of one HD DVB-T
> recording from my channels, I seem to need about 10 Mbytes/s or 80
> Mbit/s for one recording.  So being generous and allocating 100
> Mbit/s, you would think that a 1 Gbit/s Ethernet would be fine for up
> to 10 recordings at once.  But that ignores other traffic, and unless
> the NFS recording traffic has its packets flagged as high priority and
> the switch between the backend PC and the NFS box does QoS and gives
> those packets priority over other traffic, then you will have a
> problem with just one recording if something else is using all the
> bandwidth it can get.  Just you copying a file to the NFS box from
> another device can use all the available bandwidth on the NFS box's
> one Ethernet port, and has the potential to cause gaps in your
> recordings.  Doing backups to the NFS box (a common use for such a
> box) is very likely to cause trouble unless the backup software has
> options to be set to use lower bandwidth, or can be set to only do
> backups when there will never be any recordings happening.
>

Thank you - def going to work on this - i may move my hard drive on my
"old" myth BE (which is what is currently working as the NFS server) to the
new BE and thus make it local. However, for now, I am ONLY doing the Videos
part, not recording any TV shows. My backend is currently a VM so the
antenna is an issue. But I digress.


>
> So I would suggest doing some serious testing to see if it is OK to
> record directly to the NFS box before setting it up that way.  How
> many recordings at once do think you will be doing?  How many tuners
> do you have?  How many multirec recordings will each tuner do at once?
>
> An alternative is to record to a local drive on the mythbackend box,
> and when each recording completes (or later), move it over to the NFS
> box.  To make that work, you set up another storage group (other than
> "Default") and put the NFS recording directories in that storage
> group.  I have one like that called "Archives", for my 8 Tbyte
> shingled drives that can not be used for recordings as they can just
> stop for several seconds while they do a shingle re-write operation.
> Then you put the local hard drive directories in the "Default" storage
> group, and if you never create any recording rules that use the
> "Archives" storage group, nothing will ever be recorded to that
> storage group, but anything you move into the "Archives" storage group
> will be visible to mythbackend and can be played and deleted.
>
> I have written myself some Python for moving files to my archive
> drives.  It keeps a check on MythTV activity and stops moving files
> when a recording starts or just before the next one is scheduled to
> start.  It is probably not quite what you would want for moving files
> from your backend local drive(s) to NFS, but I would be happy to
> modify it to do what you need if you decide to go that way.
>
> Even with a local recording drive, if it is spinning rust I use a rule
> of thumb that there should be no more than three recordings going to a
> drive at once.  The problem is about the time taken for head movements
> between the recording files, and also to the directory areas when the
> file needs to be expanded.  And it is even worse if your system and
> database are on the same drive.  Then I would do no more than two
> recordings at once.  Modern hard drives have very fast write speeds,
> but the head movement has not sped up nearly as much as the on-track
> write speed has, so it is easy to get fooled by the fast write speed
> in the specification.
>
> Of course, if the local drive is an SSD, then you can usually write to
> many more files at once as there is no head movement.  All you have to
> watch out for there is the problem of the erase times.  If the spare
> space on an SSD has already been erased, its write times are very
> fast.  But if the drive is using a weekly TRIM operation to do the
> erasing (which is the default in Ubuntu), then the large file sizes of
> recording files can cause it to run out of erased blocks and have to
> do a block erase before each block write.  That makes an SSD write
> very slowly.  So if you record to SSD you should either be using the
> "discard" option on the SSD when it is mounted from fstab, or running
> the fstrim cron job much more often.  I run my fstrim cron job daily,
> but I do not record to the SSD - it just gets a lot of database
> activity.
>


What about all these issues - the Myth Proto Issue. I notice that this
comes up randomly every other day as if telling me that the BE is busy. But
a 8 Gb server with 8 Gb swap, only being a backend, nothing else, hard to
think it is so busy it can't let a frontend talk to it.

https://pastebin.com/Sq8Bp8yc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20180514/ef29058f/attachment.html>


More information about the mythtv-users mailing list