<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Jan 4, 2014 at 4:08 PM, Jim Stichnoth <span dir="ltr"><<a href="mailto:stichnot@gmail.com" target="_blank">stichnot@gmail.com</a>></span> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sat, Jan 4, 2014 at 10:11 AM, Craig Cook <span dir="ltr"><<a href="mailto:cncook001@yahoo.com" target="_blank">cncook001@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>>A big problem I have is that when the<br>
<br>
>server reboots, the clients basically lose their connection to the server<br>
>and fail to reestablish it, and all the clients ultimately have to be<br>
>hard-rebooted.<br>
<br>
</div>I just tested this. LTSP frontend running mythtv. Rebooted my backend. Tried to use the frontend after backend was running and mythtv client crashed. Mouse still worked. Left me at a desktop. Desktop didn't work though, had to reboot diskless PC to get it connected again.<br>
<br>
I have used minimyth and it does not have this issue (as long as you don't try and use the client while the backend is down).<br><br></blockquote></span><div>Thanks for checking. This pretty much matches what I see on 10.04. My sense is that files, applications, libraries. etc., that happen to be resident in memory will still work, but new accesses fail until the client reboots.</div>
<div><br></div><div>When I looked into this problem earlier, I found that nbd-client has a "-persist" option which seems like what is needed here. I played around with that, but I can't figure out how to actually get nbd-client started with "-persist" on the client. I added "-persist" to /etc/init.d/nbd-client, but probably nbd-client is being started in some other way, as "-persist" doesn't show up when looking at the command line through the ps command.</div><span><font color="#888888">
<div><br></div><div>Jim</div></font></span></div></div></div>
</blockquote></div><br></div><div class="gmail_extra"><div class="gmail_quote">Sorry for the long necro-post, but I think this might be useful for others. References:</div><div class="gmail_quote"><a href="http://www.gossamer-threads.com/lists/mythtv/users/560670?do=post_view_threaded#560670" target="_blank">http://www.gossamer-threads.com/lists/mythtv/users/560670?do=post_view_threaded#560670</a> [my original question in this thread]<br></div><div class="gmail_quote"><a href="http://www.gossamer-threads.com/lists/mythtv/users/469540?do=post_view_flat#469540" target="_blank">http://www.gossamer-threads.com/lists/mythtv/users/469540?do=post_view_flat#469540</a> [my even older question I hoped this thread would solve]<br></div><div class="gmail_quote"><br></div><div class="gmail_quote">I recently set up a new server to replace my existing 10.04 server that doubles as a master backend and a server for 4 PXE-booting diskless frontends. As part of setting up the fat-client environment for the frontends, I wanted to solve the problem where I am forced to hard-reboot the 4 frontends every time the server reboots. I think I pretty much solved it now. Here is a description of what's going on, and my solution.</div><div class="gmail_quote"><br></div><div class="gmail_quote">The way 10.04 mythbuntu-diskless sets up the client file system is a bit convoluted. Start with a read-only base file system that is shared among all clients. Running "ltsp-update-image" on the server takes the staged file system from /opt/ltsp/i386 or /opt/ltsp/amd64 and converts it to a single file which represents a "squashfs" file system. This file, which may be named i386.img or amd64.img, is exported using the NBD protocol, via the nbd-server process. The diskless client uses nbd-client to map the remote .img file to something like /dev/nbd0, and then it mounts it somewhere like /rofs (translation: read-only file system) using a command like:</div><div class="gmail_quote"> mount -o ro -t squashfs /dev/nbd0 /rofs</div><div class="gmail_quote"><br></div><div class="gmail_quote">Next, the diskless client NFS-mounts its own customized read-write overlay, onto somewhere like /cow (translation: copy-on-write) using the "nfsmount" program included in the initramfs image.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Finally, /rofs and /cow are joined together into the / (root) file system via the AUFS file system, such that any modifications to files in the read-only base end up being copied into the read-write file system. This is a nice setup in that changes made by the client persist across reboots.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Now, what happens when the server reboots? For the NFS overlay, it actually works pretty well. A process accessing a file may hang until the NFS server comes back, but then it reconnects and things continue. For the NBD-based file system, not so good. From what I can tell, the nbd-client process just dies, even if nbd-client is started with the "-persist" flag. Maybe the nbd-proxy program would help with resilience here, but web searches indicate problems with nbd-proxy, at least in the past, resulting in distros disabling its use. So the result of a server reboot is a dead nbd-client, and it seems there's nothing to be done except reboot the clients.</div><div class="gmail_quote"><br></div><div class="gmail_quote">But it gets worse, kind of. If you want to make changes on the server to benefit all clients, you do it maybe in a /opt/ltsp/amd64 chroot, and then re-image using ltsp-update-image. This is almost certain to mess up the base mount on the clients and require them to be rebooted. Fortunately, this seems to be controllable - log into each client and run "sudo shutdown -r +5" to reboot after a 5-minute delay, run ltsp-update-image on the server, and wait for the reboots. (Doing something like "sleep 300 ; reboot" on the client may not work, as it may not be able to find the "reboot" binary after the base image changes.) Still, it seems a lot more painful than it ought to be.</div><div class="gmail_quote"><br></div><div class="gmail_quote">The solution I like best is to directly use NFS-mounted, read-only /opt/ltsp/amd64 as the base file system, combined with the read-write NFS overlay. This takes nbd-client out of the equation, as well as the issues with updating the squashfs base file system. The disadvantage is that the client seems to take a few seconds longer to boot, but that's fine with me given the resilience benefits (and the fact that I very rarely reboot the clients).</div><div class="gmail_quote"><br></div><div class="gmail_quote">On 10.04 mythbuntu-diskless, I got this working by modifying file scripts/mythbuntu_nbd in the initrd.img file - changed this line:</div><div class="gmail_quote"> nbd-client ${NBD_ROOT_SERVER} ${NBD_ROOT_PORT} /dev/nbd0 -persist && mount -o ro -t squashfs /dev/nbd0 /rofs<br></div><div class="gmail_quote">to become:</div><div class="gmail_quote"> nfsmount -o nolock -o ro 192.168.0.205:/opt/ltsp/i386 /rofs<br></div><div class="gmail_quote">Note that 192.168.0.205 is the server address, and this setup uses i386 as opposed to amd64. This also required the server to NFS-export /opt/ltsp/i386 read-only. Done cleanly, these would be parameters passed on the boot line or inferred from the environment (and also avoid setting up /dev/nbd0 in the first place), but this was fine as a proof of concept.</div><div class="gmail_quote"><br></div><div class="gmail_quote">I rebooted the client machines with this configuration, and then rebooted the server. Maybe 30 seconds after the server came up, the clients were back to normal without needing to be rebooted. One of the clients had its audio flake out for some reason, but it was fine after restarting mythfrontend.</div><div class="gmail_quote"><br></div><div class="gmail_quote">With that successful outcome, I started looking into the diskless options in 14.04. It doesn't look like 14.04 offers anything like the 10.04 mythbuntu-diskless method of using a persistent NFS file system for the copy-on-write overlay. Looking at the LTSP option (ltsp-server-standalone package), there is an option to use a non-persistent ramfs for the copy-on-write portion. There is also an option to mount the squashfs directly over NFS rather than going through NBD. Neither option is ideal. Also, I feel that my needs don't quite align with the LTSP model and I have to work too hard to maintain consistent passwd file, hosts file, etc. across all the clients.</div><div class="gmail_quote"><br></div><div class="gmail_quote">In the end, I set up my own diskless NFS/AUFS environment based on stock Ubuntu 14.04, as follows. (If I weren't in the habit of building MythTV myself, I would probably base it on Mythbuntu instead.) Sorry for the novelette, I didn't realize it was going to be this long...</div><div class="gmail_quote"><br></div><div class="gmail_quote">1. Boot off an Ubuntu installation ISO and partition the hard drive. I partitioned as follows:</div><div class="gmail_quote"><div class="gmail_quote"> 100MB EFI</div><div class="gmail_quote"> 50,000MB /</div><div class="gmail_quote"> 50,000MB /frontend-rofs</div><div class="gmail_quote"> 50,000MB /frontend-overlay</div><div class="gmail_quote"> 100,000MB /logs</div><div class="gmail_quote"> 25,000MB swap</div><div class="gmail_quote"> remainder /storage</div><div class="gmail_quote">A very generous 50GB for the server root file system, another 50GB for the client read-only root, and a further 50GB for the various client read-write overlays. I also made a separate /logs partition for MythTV log files, because in the past I've had the frontend or backend run amok and fill up my root file system, which is always interesting to debug.</div><div class="gmail_quote"><br></div><div class="gmail_quote">2. Get Ubuntu to install entirely onto the /frontend-rofs partition. Make sure it ignores and doesn't try to mount any of the other partitions, including the swap partition, since these ultimately won't be available on the client.</div><div class="gmail_quote"><br></div><div class="gmail_quote">3. Boot off this partition. Add a mythtv user and a mythtv group. (I used vipw and vigr to set the uid and gid to the same values as the system I was upgrading from to minimize permission issues, then "chown -R mythtv.mythtv ~mythtv"). Login as mythtv and do all the GUI setup you can think of from the client perspective, such as enable auto-login, disable screensaver, etc.</div><div class="gmail_quote"><br></div><div class="gmail_quote">4. Once that all looks good, reboot off the Ubuntu installation ISO, don't change the existing partitions but do add the mount points, and let it install a second time. Repeat #3 but customize from the server perspective. (I made the mistake of first installing the server and then installing the client, and then had to do some panicked work to get grub to boot off the / file system instead of the /frontend-rofs file system.) (Also note: it might be reasonable to do just the server OS installation and then clone the / partition onto the /frontend-rofs partition using rsync.)</div><div class="gmail_quote"><br></div><div class="gmail_quote">5. Configure the server to NFS-export /frontend/rofs read-only, and /frontend-overlay and /logs as read-write. Set up PXE booting, dhcp server, tftp server, as described in a multitude of places.</div><div class="gmail_quote"><br></div><div class="gmail_quote">6. Create a helper script called "fe_chroot":</div><div class="gmail_quote"><div class="gmail_quote">#!/bin/sh</div><div class="gmail_quote">sudo mount -o bind /dev /frontend-rofs/dev<br></div><div class="gmail_quote">sudo mount -o bind /sys /frontend-rofs/sys</div><div class="gmail_quote">sudo mount -o bind /run /frontend-rofs/run</div><div class="gmail_quote">sudo mount -t proc /proc /frontend-rofs/proc</div><div class="gmail_quote">sudo chroot /frontend-rofs /bin/bash<br></div><div class="gmail_quote">sudo umount /frontend-rofs/dev<br></div><div class="gmail_quote">sudo umount /frontend-rofs/sys</div><div class="gmail_quote">sudo umount /frontend-rofs/run</div><div class="gmail_quote">sudo umount /frontend-rofs/proc</div><div><br></div></div><div class="gmail_quote">7. Do the following three commands (note: I'm not sure syslinux is actually needed):</div><div class="gmail_quote">fe_chroot</div><div class="gmail_quote">apt-get install syslinux initramfs-tools<br></div><div class="gmail_quote">exit</div><div class="gmail_quote"><br></div><div class="gmail_quote">8. There should now be a directory /frontend-rofs/etc/initramfs-tools/. Execute the command:</div><div class="gmail_quote">echo aufs >> /frontend-rofs/etc/initramfs-tools/modules</div><div class="gmail_quote">Create a file in /frontend-rofs/etc/initramfs-tools/scripts/init-bottom/ called 00_overlay, and "chmod +x 00_overlay", with following contents (not including the ===== lines of course).</div><div class="gmail_quote"><div class="gmail_quote">================================================================</div><div class="gmail_quote">#!/bin/sh</div><div class="gmail_quote"><br></div><div class="gmail_quote"># This script's home location is:</div><div class="gmail_quote"># /etc/initramfs-tools/scripts/init-bottom/00_overlay</div><div class="gmail_quote"># Also modify /etc/initramfs-tools/modules to add aufs.</div><div class="gmail_quote"><br></div><div class="gmail_quote">PREREQ=""</div><div class="gmail_quote">prereqs()</div><div class="gmail_quote">{</div><div class="gmail_quote"> echo "$PREREQ"</div><div class="gmail_quote">}</div><div class="gmail_quote"><br></div><div class="gmail_quote">case "$1" in</div><div class="gmail_quote">prereqs)</div><div class="gmail_quote"> prereqs</div><div class="gmail_quote"> exit 0</div><div class="gmail_quote"> ;;</div><div class="gmail_quote">esac</div><div class="gmail_quote"><br></div><div class="gmail_quote">. /scripts/functions</div><div class="gmail_quote"><br></div><div class="gmail_quote">overlay=</div><div class="gmail_quote">hostname="client"</div><div class="gmail_quote"><br></div><div class="gmail_quote">for x in $(cat /proc/cmdline); do</div><div class="gmail_quote"> case $x in</div><div class="gmail_quote"> overlay=*)</div><div class="gmail_quote"> overlay=${x#overlay=}</div><div class="gmail_quote"> ;;</div><div class="gmail_quote"> hostname=*)</div><div class="gmail_quote"> hostname=${x#hostname=}</div><div class="gmail_quote"> ;;</div><div class="gmail_quote"> esac</div><div class="gmail_quote">done</div><div class="gmail_quote"><br></div><div class="gmail_quote"># Don't create the overlay if ${overlay} is unset.</div><div class="gmail_quote">[ -z "${overlay}" ] && exit 0</div><div class="gmail_quote"><br></div><div class="gmail_quote"># Leverage the existing /scripts/nfs to mount the NFS overlay, by</div><div class="gmail_quote"># setting various envvars in a subshell and calling the script.</div><div class="gmail_quote"># Usage: mount_nfs_overlay remote_src local_target</div><div class="gmail_quote">mount_nfs_overlay()</div><div class="gmail_quote">{(</div><div class="gmail_quote"> NFSROOT="$1"</div><div class="gmail_quote"> rootmnt="$2"</div><div class="gmail_quote"> readonly=n</div><div class="gmail_quote"> ROOTDELAY=10 # don't want to wait 180s on failure</div><div class="gmail_quote"> . /scripts/nfs</div><div class="gmail_quote"> mountroot</div><div class="gmail_quote">)}</div><div class="gmail_quote"><br></div><div class="gmail_quote">modprobe aufs || panic "Unable to load aufs module"</div><div class="gmail_quote">mkdir -p /rofs /cow || panic "mkdir /rofs /cow failed"</div><div class="gmail_quote">case "${overlay}" in</div><div class="gmail_quote"> *:*)</div><div class="gmail_quote"> mount_nfs_overlay "${overlay}" /cow \</div><div class="gmail_quote"> || panic "nfsmount failed"</div><div class="gmail_quote"> ;;</div><div class="gmail_quote"> UUID=*)</div><div class="gmail_quote"> mount "/dev/disk/by-uuid/${overlay#UUID=}" /cow \</div><div class="gmail_quote"> || panic "mount ${overlay} failed"</div><div class="gmail_quote"> ;;</div><div class="gmail_quote"> *)</div><div class="gmail_quote"> mount "${overlay}" /cow \</div><div class="gmail_quote"> || panic "mount ${overlay} failed"</div><div class="gmail_quote"> ;;</div><div class="gmail_quote">esac</div><div class="gmail_quote">mkdir -p /cow/${hostname} || panic "mkdir /cow/hostname=${hostname} failed"</div><div class="gmail_quote">mount -o move ${rootmnt} /rofs || panic "move ${rootmnt} to /rofs failed"</div><div class="gmail_quote">mount -t aufs -o br=/cow/${hostname}=rw:/rofs=ro,xino=/run/aufs.xino none \<br></div><div class="gmail_quote"> ${rootmnt} || panic "aufs mount to ${rootmnt} failed"</div><div class="gmail_quote">mkdir -p ${rootmnt}/rofs \</div><div class="gmail_quote"> || panic "mkdir /rootmnt=${rootmnt}/rofs failed"</div><div class="gmail_quote">mount -o move /rofs ${rootmnt}/rofs \</div><div class="gmail_quote"> || panic "move /rofs to rootmnt=${rootmnt}/rofs failed"</div><div class="gmail_quote">mkdir -p ${rootmnt}/cow \</div><div class="gmail_quote"> || panic "mkdir rootmnt=${rootmnt}/cow failed"</div><div class="gmail_quote">mount -o bind /cow/${hostname} ${rootmnt}/cow \</div><div class="gmail_quote"> || panic "bind /cow/hostname=${hostname} to rootmnt=${rootmnt}/cow failed"</div><div class="gmail_quote">echo "${hostname}" > ${rootmnt}/etc/hostname</div><div class="gmail_quote">================================================================</div><div><br></div></div><div>9. Build and install a new diskless client initrd image:</div><div>fe_chroot</div><div>mkinitramfs -o /boot/initrd.img.overlay</div><div>exit</div><div>cp /frontend-rofs/boot/initrd.img.overlay /var/lib/tftpboot/</div><div><br></div><div>10. SUPER IMPORTANT: Disable NetworkManager on the client:</div><div>echo "manual" > /frontend-rofs/etc/init/network-manager.override<br></div><div>Otherwise NetworkManager will disable the network while booting, then NFS goes away, and the boot hangs, very hard to diagnose the problem.</div><div><br></div><div>11. The appropriate files in /var/lib/tftpboot/pxelinux.cfg/ should have this APPEND line:</div><div>APPEND hostname=my_client_name blacklist=nouveau ro boot=nfs nfsroot=192.168.0.216:/frontend-rofs overlay=192.168.0.216:/frontend-overlay initrd=initrd.img.overlay ip=dhcp debug<br></div><div>Instead of 192.168.0.216, use the server's fixed IP address. Instead of my_client_name, use the hostname that you want for that particular client. I'm not sure exactly what circumstances blacklist=nouveau is needed.</div><div>The cool thing here is that you can use this for all 6 configurations: root on local disk and no overlay; root on NFS and no overlay; root on local disk with local disk overlay; root on local disk with NFS overlay; root on NFS with local disk overlay; root on NFS with NFS overlay. (Yes, I tested all 6.)</div><div><br></div><div>12. PXE-boot the client(s). Hopefully it comes up with a normal desktop environment.</div><div><br></div><div>Any local file changes made on a client can be viewed on the server in /frontend-overlay/`hostname`/. If any changes need to be made for all clients, make them within the /frontend-rofs/ directory, or (e.g. to "apt-get install") run fe_chroot and execute "directly". This is really great in my setup where there are 4 mostly identical frontends. Sometimes these changes confuse the client, but you can usually recover on the client by running "mount -o remount /" on the client to avoid rebooting.</div><div><br></div><div>In my setup, I can "rm -rf /frontend-overlay/<hostname>" and reboot the client, and it will automagically re-synthesize its environment. So far I have just one place with customizations based on the client's hostname (overriding the session/window/whatever manager's choice of screen resolution).</div><div><br></div><div>Of course, after completing the above steps and successfully net-booting, there's a lot more setup work to be done, but at least there's a very nice distinction between work done for the server and work done for the shared client space.</div><div><br></div><div>There's one important subtlety I should point out. Step #9 builds the client's initrd image within the /frontend-rofs/ file system and then copies it into the (server's) root file system to be served up under tftp, even though the natural inclination may be to build it within the root file system. (Specifically, I'm referring to the two separate copies of /etc/initramfs-tools/, one under / and the other under /frontend-rofs/.) This is because some "apt-get install" operations for the client want to make changes to the initrd image and rebuild it (notably, installing proprietary nvidia drivers). Whenever this happens, step #9 needs to be repeated, and so to get the benefit of both the package changes and the 00_overlay script, the 00_overlay script needs to live there. It would be nice if /var/lib/tftpboot/initrd.img.overlay were more tightly linked to a suitable /frontend-rofs/boot/initrd.img, but I haven't come up with a good way (and also, it's not that bothersome so I'm not highly motivated).</div><div><br></div><div>Anyway, thanks for listening, and I hope this helps others in the future. :)</div><div><br></div><div>Jim</div></div></div></div>