[mythtv-users] Diskless LTSP Frontend

[Jim Stichnoth]

On Sat, Jan 4, 2014 at 4:08 PM, Jim Stichnoth <stichnot at gmail.com> wrote:

> On Sat, Jan 4, 2014 at 10:11 AM, Craig Cook <cncook001 at yahoo.com> wrote:
>
>> >A big problem I have is that when the
>>
>> >server reboots, the clients basically lose their connection to the server
>> >and fail to reestablish it, and all the clients ultimately have to be
>> >hard-rebooted.
>>
>> I just tested this.  LTSP frontend running mythtv.  Rebooted my backend.
>> Tried to use the frontend after backend was running and mythtv client
>> crashed.  Mouse still worked. Left me at a desktop.  Desktop didn't work
>> though, had to reboot diskless PC to get it connected again.
>>
>> I have used minimyth and it does not have this issue (as long as you
>> don't try and use the client while the backend is down).
>>
>> Thanks for checking.  This pretty much matches what I see on 10.04.  My
> sense is that files, applications, libraries. etc., that happen to be
> resident in memory will still work, but new accesses fail until the client
> reboots.
>
> When I looked into this problem earlier, I found that nbd-client has a
> "-persist" option which seems like what is needed here.  I played around
> with that, but I can't figure out how to actually get nbd-client started
> with "-persist" on the client.  I added "-persist" to
> /etc/init.d/nbd-client, but probably nbd-client is being started in some
> other way, as "-persist" doesn't show up when looking at the command line
> through the ps command.
>
> Jim
>

Sorry for the long necro-post, but I think this might be useful for
others.  References:
http://www.gossamer-threads.com/lists/mythtv/users/560670?do=post_view_threaded#560670
[my original question in this thread]
http://www.gossamer-threads.com/lists/mythtv/users/469540?do=post_view_flat#469540
[my even older question I hoped this thread would solve]

I recently set up a new server to replace my existing 10.04 server that
doubles as a master backend and a server for 4 PXE-booting diskless
frontends.  As part of setting up the fat-client environment for the
frontends, I wanted to solve the problem where I am forced to hard-reboot
the 4 frontends every time the server reboots.  I think I pretty much
solved it now.  Here is a description of what's going on, and my solution.

The way 10.04 mythbuntu-diskless sets up the client file system is a bit
convoluted.  Start with a read-only base file system that is shared among
all clients.  Running "ltsp-update-image" on the server takes the staged
file system from /opt/ltsp/i386 or /opt/ltsp/amd64 and converts it to a
single file which represents a "squashfs" file system.  This file, which
may be named i386.img or amd64.img, is exported using the NBD protocol, via
the nbd-server process.  The diskless client uses nbd-client to map the
remote .img file to something like /dev/nbd0, and then it mounts it
somewhere like /rofs (translation: read-only file system) using a command
like:
  mount -o ro -t squashfs /dev/nbd0 /rofs

Next, the diskless client NFS-mounts its own customized read-write overlay,
onto somewhere like /cow (translation: copy-on-write) using the "nfsmount"
program included in the initramfs image.

Finally, /rofs and /cow are joined together into the / (root) file system
via the AUFS file system, such that any modifications to files in the
read-only base end up being copied into the read-write file system.  This
is a nice setup in that changes made by the client persist across reboots.

Now, what happens when the server reboots?  For the NFS overlay, it
actually works pretty well.  A process accessing a file may hang until the
NFS server comes back, but then it reconnects and things continue.  For the
NBD-based file system, not so good.  From what I can tell, the nbd-client
process just dies, even if nbd-client is started with the "-persist" flag.
Maybe the nbd-proxy program would help with resilience here, but web
searches indicate problems with nbd-proxy, at least in the past, resulting
in distros disabling its use.  So the result of a server reboot is a dead
nbd-client, and it seems there's nothing to be done except reboot the
clients.

But it gets worse, kind of.  If you want to make changes on the server to
benefit all clients, you do it maybe in a /opt/ltsp/amd64 chroot, and then
re-image using ltsp-update-image.  This is almost certain to mess up the
base mount on the clients and require them to be rebooted.  Fortunately,
this seems to be controllable - log into each client and run "sudo shutdown
-r +5" to reboot after a 5-minute delay, run ltsp-update-image on the
server, and wait for the reboots.  (Doing something like "sleep 300 ;
reboot" on the client may not work, as it may not be able to find the
"reboot" binary after the base image changes.)  Still, it seems a lot more
painful than it ought to be.

The solution I like best is to directly use NFS-mounted, read-only
/opt/ltsp/amd64 as the base file system, combined with the read-write NFS
overlay.  This takes nbd-client out of the equation, as well as the issues
with updating the squashfs base file system.  The disadvantage is that the
client seems to take a few seconds longer to boot, but that's fine with me
given the resilience benefits (and the fact that I very rarely reboot the
clients).

On 10.04 mythbuntu-diskless, I got this working by modifying file
scripts/mythbuntu_nbd in the initrd.img file - changed this line:
  nbd-client ${NBD_ROOT_SERVER} ${NBD_ROOT_PORT} /dev/nbd0 -persist &&
mount -o ro -t squashfs /dev/nbd0 /rofs
to become:
  nfsmount -o nolock -o ro 192.168.0.205:/opt/ltsp/i386 /rofs
Note that 192.168.0.205 is the server address, and this setup uses i386 as
opposed to amd64.  This also required the server to NFS-export
/opt/ltsp/i386 read-only.  Done cleanly, these would be parameters passed
on the boot line or inferred from the environment (and also avoid setting
up /dev/nbd0 in the first place), but this was fine as a proof of concept.

I rebooted the client machines with this configuration, and then rebooted
the server.  Maybe 30 seconds after the server came up, the clients were
back to normal without needing to be rebooted.  One of the clients had its
audio flake out for some reason, but it was fine after restarting
mythfrontend.

With that successful outcome, I started looking into the diskless options
in 14.04.  It doesn't look like 14.04 offers anything like the 10.04
mythbuntu-diskless method of using a persistent NFS file system for the
copy-on-write overlay.  Looking at the LTSP option (ltsp-server-standalone
package), there is an option to use a non-persistent ramfs for the
copy-on-write portion.  There is also an option to mount the squashfs
directly over NFS rather than going through NBD.  Neither option is ideal.
Also, I feel that my needs don't quite align with the LTSP model and I have
to work too hard to maintain consistent passwd file, hosts file, etc.
across all the clients.

In the end, I set up my own diskless NFS/AUFS environment based on stock
Ubuntu 14.04, as follows.  (If I weren't in the habit of building MythTV
myself, I would probably base it on Mythbuntu instead.)  Sorry for the
novelette, I didn't realize it was going to be this long...

1. Boot off an Ubuntu installation ISO and partition the hard drive.  I
partitioned as follows:
      100MB EFI
   50,000MB /
   50,000MB /frontend-rofs
   50,000MB /frontend-overlay
  100,000MB /logs
   25,000MB swap
  remainder /storage
A very generous 50GB for the server root file system, another 50GB for the
client read-only root, and a further 50GB for the various client read-write
overlays.  I also made a separate /logs partition for MythTV log files,
because in the past I've had the frontend or backend run amok and fill up
my root file system, which is always interesting to debug.

2. Get Ubuntu to install entirely onto the /frontend-rofs partition.  Make
sure it ignores and doesn't try to mount any of the other partitions,
including the swap partition, since these ultimately won't be available on
the client.

3. Boot off this partition.  Add a mythtv user and a mythtv group.  (I used
vipw and vigr to set the uid and gid to the same values as the system I was
upgrading from to minimize permission issues, then "chown -R mythtv.mythtv
~mythtv").  Login as mythtv and do all the GUI setup you can think of from
the client perspective, such as enable auto-login, disable screensaver, etc.

4. Once that all looks good, reboot off the Ubuntu installation ISO, don't
change the existing partitions but do add the mount points, and let it
install a second time.  Repeat #3 but customize from the server
perspective.  (I made the mistake of first installing the server and then
installing the client, and then had to do some panicked work to get grub to
boot off the / file system instead of the /frontend-rofs file system.)
 (Also note: it might be reasonable to do just the server OS installation
and then clone the / partition onto the /frontend-rofs partition using
rsync.)

5. Configure the server to NFS-export /frontend/rofs read-only, and
/frontend-overlay and /logs as read-write.  Set up PXE booting, dhcp
server, tftp server, as described in a multitude of places.

6. Create a helper script called "fe_chroot":
#!/bin/sh
sudo mount -o bind /dev /frontend-rofs/dev
sudo mount -o bind /sys /frontend-rofs/sys
sudo mount -o bind /run /frontend-rofs/run
sudo mount -t proc /proc /frontend-rofs/proc
sudo chroot /frontend-rofs /bin/bash
sudo umount /frontend-rofs/dev
sudo umount /frontend-rofs/sys
sudo umount /frontend-rofs/run
sudo umount /frontend-rofs/proc

7. Do the following three commands (note: I'm not sure syslinux is actually
needed):
fe_chroot
apt-get install syslinux initramfs-tools
exit

8. There should now be a directory /frontend-rofs/etc/initramfs-tools/.
Execute the command:
echo aufs >> /frontend-rofs/etc/initramfs-tools/modules
Create a file in /frontend-rofs/etc/initramfs-tools/scripts/init-bottom/
called 00_overlay, and "chmod +x 00_overlay", with following contents (not
including the ===== lines of course).
================================================================
#!/bin/sh

# This script's home location is:
# /etc/initramfs-tools/scripts/init-bottom/00_overlay
# Also modify /etc/initramfs-tools/modules to add aufs.

PREREQ=""
prereqs()
{
     echo "$PREREQ"
}

case "$1" in
prereqs)
     prereqs
     exit 0
     ;;
esac

. /scripts/functions

overlay=
hostname="client"

for x in $(cat /proc/cmdline); do
    case $x in
        overlay=*)
            overlay=${x#overlay=}
            ;;
        hostname=*)
            hostname=${x#hostname=}
            ;;
    esac
done

# Don't create the overlay if ${overlay} is unset.
[ -z "${overlay}" ] && exit 0

# Leverage the existing /scripts/nfs to mount the NFS overlay, by
# setting various envvars in a subshell and calling the script.
# Usage: mount_nfs_overlay remote_src local_target
mount_nfs_overlay()
{(
        NFSROOT="$1"
        rootmnt="$2"
        readonly=n
        ROOTDELAY=10 # don't want to wait 180s on failure
        . /scripts/nfs
        mountroot
)}

modprobe aufs || panic "Unable to load aufs module"
mkdir -p /rofs /cow || panic "mkdir /rofs /cow failed"
case "${overlay}" in
    *:*)
        mount_nfs_overlay "${overlay}" /cow \
            || panic "nfsmount failed"
        ;;
    UUID=*)
        mount "/dev/disk/by-uuid/${overlay#UUID=}" /cow \
            || panic "mount ${overlay} failed"
        ;;
    *)
        mount "${overlay}" /cow \
            || panic "mount ${overlay} failed"
        ;;
esac
mkdir -p /cow/${hostname} || panic "mkdir /cow/hostname=${hostname} failed"
mount -o move ${rootmnt} /rofs || panic "move ${rootmnt} to /rofs failed"
mount -t aufs -o br=/cow/${hostname}=rw:/rofs=ro,xino=/run/aufs.xino none \
    ${rootmnt} || panic "aufs mount to ${rootmnt} failed"
mkdir -p ${rootmnt}/rofs \
    || panic "mkdir /rootmnt=${rootmnt}/rofs failed"
mount -o move /rofs ${rootmnt}/rofs \
    || panic "move /rofs to rootmnt=${rootmnt}/rofs failed"
mkdir -p ${rootmnt}/cow \
    || panic "mkdir rootmnt=${rootmnt}/cow failed"
mount -o bind /cow/${hostname} ${rootmnt}/cow \
    || panic "bind /cow/hostname=${hostname} to rootmnt=${rootmnt}/cow
failed"
echo "${hostname}" > ${rootmnt}/etc/hostname
================================================================

9. Build and install a new diskless client initrd image:
fe_chroot
mkinitramfs -o /boot/initrd.img.overlay
exit
cp /frontend-rofs/boot/initrd.img.overlay /var/lib/tftpboot/

10. SUPER IMPORTANT: Disable NetworkManager on the client:
echo "manual" > /frontend-rofs/etc/init/network-manager.override
Otherwise NetworkManager will disable the network while booting, then NFS
goes away, and the boot hangs, very hard to diagnose the problem.

11. The appropriate files in /var/lib/tftpboot/pxelinux.cfg/ should have
this APPEND line:
APPEND hostname=my_client_name blacklist=nouveau ro boot=nfs
nfsroot=192.168.0.216:/frontend-rofs overlay=192.168.0.216:/frontend-overlay
initrd=initrd.img.overlay ip=dhcp debug
Instead of 192.168.0.216, use the server's fixed IP address.  Instead of
my_client_name, use the hostname that you want for that particular client.
I'm not sure exactly what circumstances blacklist=nouveau is needed.
The cool thing here is that you can use this for all 6 configurations: root
on local disk and no overlay; root on NFS and no overlay; root on local
disk with local disk overlay; root on local disk with NFS overlay; root on
NFS with local disk overlay; root on NFS with NFS overlay.  (Yes, I tested
all 6.)

12. PXE-boot the client(s).  Hopefully it comes up with a normal desktop
environment.

Any local file changes made on a client can be viewed on the server in
/frontend-overlay/`hostname`/.  If any changes need to be made for all
clients, make them within the /frontend-rofs/ directory, or (e.g. to
"apt-get install") run fe_chroot and execute "directly".  This is really
great in my setup where there are 4 mostly identical frontends.  Sometimes
these changes confuse the client, but you can usually recover on the client
by running "mount -o remount /" on the client to avoid rebooting.

In my setup, I can "rm -rf /frontend-overlay/<hostname>" and reboot the
client, and it will automagically re-synthesize its environment.  So far I
have just one place with customizations based on the client's hostname
(overriding the session/window/whatever manager's choice of screen
resolution).

Of course, after completing the above steps and successfully net-booting,
there's a lot more setup work to be done, but at least there's a very nice
distinction between work done for the server and work done for the shared
client space.

There's one important subtlety I should point out.  Step #9 builds the
client's initrd image within the /frontend-rofs/ file system and then
copies it into the (server's) root file system to be served up under tftp,
even though the natural inclination may be to build it within the root file
system.  (Specifically, I'm referring to the two separate copies of
/etc/initramfs-tools/, one under / and the other under /frontend-rofs/.)
 This is because some "apt-get install" operations for the client want to
make changes to the initrd image and rebuild it (notably, installing
proprietary nvidia drivers).  Whenever this happens, step #9 needs to be
repeated, and so to get the benefit of both the package changes and the
00_overlay script, the 00_overlay script needs to live there.  It would be
nice if /var/lib/tftpboot/initrd.img.overlay were more tightly linked to a
suitable /frontend-rofs/boot/initrd.img, but I haven't come up with a good
way (and also, it's not that bothersome so I'm not highly motivated).

Anyway, thanks for listening, and I hope this helps others in the future. :)

Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mythtv.org/pipermail/mythtv-users/attachments/20150409/89d03020/attachment.html>