[mythtv-users] recent "TFW -- took a long time" warnings

[Warpme]

On 6/23/12 4:01 AM, danielk wrote:
> On 06/22/2012 05:26 PM, Warpme wrote:
>> If anybody experiencing issue like described in my original post:
>> it is worth to check kernel VM settings regarding dirty pages swapping.
>> Pls do test with following:
>
> I haven't been seeing the TFW errors, but that the virtual memory
> settings have anything to do with it is interesting. How much
> virtual memory is your mythbackend process consuming?
>
> -- Daniel
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://www.mythtv.org/mailman/listinfo/mythtv-users
>
Hi Daniel,

Currently it is 1,67G while RSS is 122M.
I have 4G RAM, no swap. BE is only console (no GFX shell) so normally I 
have 2,4-2,7G allocated as buffers.

Some comments form my side :-)
Distribution of TFW errors (they are consolidated around short periods 
of time) and lack of any correlation with other user process activity 
directed me to dirty pages/swap area.
I was skeptic about swap as having no swap there shouldn't be aspects 
related to swapiness, but dirty pages write-out were good candidate to 
investigate.
According to various sources, when pdflush daemons are writing huge 
amounts of dirty pages (on systems with high dirty_ratio), they 
potentially can produce "write-hog" and periodically block all I/O on 
the system.
Fortunately fine tuning (exact amount, not % in installed RAM) was 
addressed in https://lkml.org/lkml/2008/11/23/160
I decided to implement strategy: lets VM dirty_pages writes-out will be 
with possible minimal amount of dirty pages as write-out is quite 
expensive from real-time point of view. Price of this strategy are 
frequent writes, but it is much less than default behavior where in 
worst case kernel potentially can write-out 40% in single period of time 
(=1,6G my case as kernel default dirty_ratio is 40%). So I set 
vm.dirty_background_bytes = 102400.

My lessons learn are following:
-If there are TFW errors, first thing worth to check is their 
correlations with other user space activities (like copying huge files, 
etc).
-If there is no visible correlation then it is worth to look on their 
distribution. If distribution is regular - probably system has issue 
with IO BW or IO subsystem has problems itself (failing drive, etc).
-If they are consolidated, then probably issue is like mine: kernel 
dirty_pages write-outs strategy. It is worth to launch:
watch -n1 grep -A 1 dirty /proc/vmstat
and look on nr_dirty variations.

-br




-------------- next part --------------
A non-text attachment was scrubbed...
Name: warpme.vcf
Type: text/x-vcard
Size: 83 bytes
Desc: not available
URL: <http://www.mythtv.org/pipermail/mythtv-users/attachments/20120624/f195bbc5/attachment.vcf>