[mythtv] libsensors making mythbackend very unstable
Andrew Meredith
andrew at anvil.org
Mon Jan 8 12:47:36 UTC 2007
Sorry for answering my own question, but I just realised what the
problem is.
The libsensors function calls are not re-entrant. They rely on a static
variable to hold their state. The initialisation call is "int
sensors_init(FILE *input);" the int is just a return code. The cleanup
call is "void sensors_cleanup(void);".
I am polling the mythbackend master from localhost and from the two
slave servers as part of the self healing system. The machines are all
tied to the ntpd clock, so are likely to start the poll very close to
each other.
Now that I have pulled the sensor kernel modules and so shortened the
time spent in this section of code, all I have done is narrow the
window. This explains both the increase in MTBF and the fact that it has
not entirely gone away.
The libsensors code is not safe in a multi-threaded environment.
It might be a plan to either remove it, or at least put a semaphore
round it.
Cheers
Andy M
Andrew Meredith wrote:
> Hi Folks,
>
> After an extended piece of debugging I finally tracked down why
> mythbackend on my mythbackend master kept falling over. It was because
> it had lm_sensors configured.
>
> It kept falling over with strange parser errors and the odd mention of
> flex. I trawled through the binaries for the source of these strings and
> failed .. I used `rpm -qa | fgrep myth` to find all myth related rpms
> and then greped them all for the strings in the mythbackend logs and
> drew a blank. I finally used ldd to fish out all the shared objects and
> grepped through them .. the strings were coming from libsensors. I used
> rpm -V to see if anything had changed in the package .. nope. Finally I
> "modprobe -r"ed the sensor modules out of the kernel and restarted
> mythbackend ... it still falls over, but now it's every 2 or 3 hours,
> rather than every 5 to 15 minutes. The weird thing is that those same
> flex related strings are still logged right before the crashes .. some
> of them at least.
>
> Is there some way that the CPU temp, and therefore the libsensors calls,
> can be made optional, post compile. I don't want to have this code in my
> execution path, but then I REALLY don't want to have to roll my own again.
>
> If you can think of any tests you would like me to run to make a bit
> more sense of this then no worries. I am quite willing to build stuff
> and even insmod the sensor related modules back in for a bit to get this
> solved. I have looked at the (svn) code and can see no good reason why
> such a simple set of library calls should cause this much grief, but
> there we are.
>
> Cheers
>
> Andy M
>
> FC5+atrms up to date
> AMD 32 bit Athlon 1GHz
> No TV cards .. that's the slaves' jobs
>
> -- /etc/rc.local --
> # I2C adapter drivers
> #modprobe i2c-viapro
> # Chip drivers
> #modprobe w83781d
> #modprobe via686a
> -------------------
>
> lm_sensors-2.10.1-45.fc5.at
> mythweather-0.20-149.fc5.at
> mythgame-0.20-149.fc5.at
> libmyth-devel-0.20-149.fc5.at
> mythtv-themes-0.20-149.fc5.at
> mythdvd-0.20-149.fc5.at
> mythbrowser-0.20-149.fc5.at
> mythtv-0.20-149.fc5.at
> mythweb-0.20-149.fc5.at
> mythgallery-0.20-149.fc5.at
> mythtv-frontend-0.20-149.fc5.at
> mythplugins-0.20-149.fc5.at
> mythtv-setup-0.20-149.fc5.at
> mytharchive-0.20-149.fc5.at
> mythmusic-0.20-149.fc5.at
> myththemes-0.20-110.at
> mythnews-0.20-149.fc5.at
> mythvideo-0.20-149.fc5.at
> mythflix-0.20-149.fc5.at
> mythtv-backend-0.20-149.fc5.at
> mythphone-0.20-149.fc5.at
> libmyth-0.20-149.fc5.at
> mythcontrols-0.20-149.fc5.at
> mythtv-suite-0.20-149.fc5.at
>
>
> Typical last log entries before failure:
>
> (My self healing scripts do a tail on the logs before starting the
> daemon back up again. I have only included the logs from dead daemon
> events, so the daemon had died of it's own accord. The test is to use ps
> to check for the presence of the PID stored in the .pid file.)
>
> ==
>
> 2007-01-08 02:58:29.458 scheduler: Scheduled items: Scheduled 1412 items
> in 21.5 = 3.78 match + 17.72 place
> 2007-01-08 02:58:29.531 Recording starts soon, AUTO-Startup assumed
> flex scanner jammed
>
> ==
>
> 2007-01-08 03:00:45.456 Canceled recording (Aborted): Frasier "Guess
> Who's Coming to Breakfast": channel 5004 on cardid 5, sourceid 5
> 2007-01-08 03:00:48.905 scheduler: Canceled recording (Aborted): Frasier
> "Guess Who's Coming to Breakfast": channel 5004 on cardid 5, sourceid 5
> 2007-01-08 03:01:07.064 Reschedule requested for id -1.
> fatal flex scanner internal error--end of buffer missed
>
> ==
>
> 2007-01-08 03:04:12.198 Connected to database 'mythconverg' at host: chaven
> Error: Line 7407: syntax error, unexpected LABEL
> Error: Line 7409: syntax error, unexpected EOL
> 2007-01-08 03:04:12.524 scheduler: Scheduled items: Scheduled 1412 items
> in 21.4 = 3.73 match + 17.69 place
> 2007-01-08 03:04:13.768 Canceled recording (Aborted): Frasier "Guess
> Who's Coming to Breakfast": channel 5004 on cardid 5, sourceid 5
> 2007-01-08 03:04:14.066 scheduler: Canceled recording (Aborted): Frasier
> "Guess Who's Coming to Breakfast": channel 5004 on cardid 5, sourceid 5
> 2007-01-08 03:04:42.028 MainServer::HandleAnnounce Monitor
> 2007-01-08 03:04:42.162 adding: coliseum.ch.anvil.org as a client
> (events: 0)
> 2007-01-08 03:04:42.332 MainServer::HandleAnnounce Monitor
> 2007-01-08 03:04:42.366 adding: coliseum.ch.anvil.org as a client
> (events: 1)
> 2007-01-08 03:05:01.063 Reschedule requested for id 0.
> 2007-01-08 03:05:33.194 Scheduled 1411 items in 32.1 = 0.07 match +
> 32.06 place
> QSocketDevice::writeBlock: Invalid socket
> 2007-01-08 03:05:42.818 scheduler: Scheduled items: Scheduled 1411 items
> in 32.1 = 0.07 match + 32.06 place
> 2007-01-08 03:05:47.178 Started recording: Frasier "Guess Who's Coming
> to Breakfast": channel 5004 on cardid 5, sourceid 5
> 2007-01-08 03:05:47.375 scheduler: Started recording: Frasier "Guess
> Who's Coming to Breakfast": channel 5004 on cardid 5, sourceid 5
> 2007-01-08 03:05:53.275 Reschedule requested for id 0.
> 2007-01-08 03:06:08.970 Scheduled 1411 items in 15.7 = 1.25 match +
> 14.44 place
> 2007-01-08 03:06:16.602 scheduler: Scheduled items: Scheduled 1411 items
> in 15.7 = 1.25 match + 14.44 place
> 2007-01-08 03:06:39.936 Reschedule requested for id -1.
> 2007-01-08 03:07:09.454 Scheduled 1411 items in 29.4 = 11.14 match +
> 18.30 place
> Error: Line 33679: Invalid keyword
> Error: Line 33684: Invalid keyword
> 2007-01-08 03:07:10.129 scheduler: Scheduled items: Scheduled 1411 items
> in 29.4 = 11.14 match + 18.30 place
> 2007-01-08 03:07:32.691 mythbackend: Running housekeeping thread
> fatal flex scanner internal error--end of buffer missed
> 2007-01-08 03:08:07.419 Unknown socket closing
>
> ==
>
> 2007-01-08 03:14:38.088 Reschedule requested for id -1.
> 2007-01-08 03:14:53.403 Scheduled 1411 items in 15.3 = 3.59 match +
> 11.72 place
> 2007-01-08 03:14:53.460 scheduler: Scheduled items: Scheduled 1411 items
> in 15.3 = 3.59 match + 11.72 place
> 2007-01-08 03:17:09.493 Reschedule requested for id -1.
> Error: Line 48142: syntax error, unexpected $end
> 2007-01-08 03:17:19.219 New DB connection, total: 5
> 2007-01-08 03:17:19.326 Connected to database 'mythconverg' at host: chaven
> 2007-01-08 03:17:19.845 mythbackend: Running housekeeping thread
> 2007-01-08 03:17:58.380 Scheduled 1411 items in 48.9 = 27.79 match +
> 21.09 place
> 2007-01-08 03:17:58.538 scheduler: Scheduled items: Scheduled 1411 items
> in 48.9 = 27.79 match + 21.09 place
> Error: Line 56270: Invalid keyword
>
>
>
>
--
____________________________________________________________
Andrew Meredith BEng CEng CITP MBCS MIET
The Anvil Organisation Ltd
Director
http://www.anvil.org/ -- sip:andrew at sip.anvil.org
andrew at anvil.org -------- +44 (0) 1249 460560
Need a Systems Administrator, just not a whole one?
Try our "As you need it" support and training services
____________________________________________________________
More information about the mythtv-dev
mailing list