[mythtv-users] 64 Bit and memory

[Richard Freeman]

Marc wrote:
> I'm running a SVN(15354) frontend/backend on Gentoo with 32 bits and I'm not
> seeing this problem.
> asgard ~ # free -m
 > snip...

I just noticed these posts.  If you're concerned about memory leaks / 
etc looking at the output of free is just about useless.  Look at how 
much memory individual processes are taking up instead.

It is completely normal for a linux system with 500GB of memory and at 
least as much hard drive space to swap to disk, and report most of the 
memory as in use just about all the time.

The reason memory seems to be always in-use is that "free" memory in 
linux is memory that is not being used AT ALL for any purpose.  Linux 
tries to minimize the amount of completely-unused memory - preferring to 
use it for buffering or cache.  When the memory is needed the cache is 
purged and the buffers are flushed.  This only improves system 
performance by reducing disk access.  If memory isn't needed for 
something else you might as well use it as cache - it doesn't cost 
anything as cache can be instantly purged to be made available.

The reason swapping happens all the time on linux is because of the 
"swapiness" setting.  If a page of memory seems to be idle for a very 
long time the system will swap it out when it is otherwise not busy to 
make more room for cache/buffers.  The assumption is that something else 
will need that memory before the idle app does, or that caching will 
generally improve performance.  So even if there is free memory the 
system will swap out empty pages.  This behavior can be tuned with a 
setting in /proc if necessary.

A memory leak is when an application allocates memory and then doesn't 
use it, or stops using memory but doesn't free it.  It causes an 
application's virtual memory use to grow, but its real memory use does 
not necessarily grow as much (because of the aforementioned swappiness - 
linux just swaps the dead pages out to disk and never reads them back). 
  The ability to swap dead pages would probably depend on whether the 
wasted memory fills entire pages - it still isn't ideal.

I haven't observed memory leaks on my 64bit gentoo system, or on my 
32-bit frontend (with about 200MB available RAM and no swap).  I have 
increased my buffer sizes on my backend, so that can consume a fair 
amount of virtual memory - but nothing really more than expected.

When my myth system is slow I've found that the usual cause is IO 
contention.  Swapping contributes to this in part, but only in part. 
The iostat tool can help troubleshoot this.  The ionice tool can help 
mitigate this.  I run the backend with a medium realtime ioniceness 
setting - this allows the backend access to the disk just about anytime 
it needs it without waiting  in line.  That and increasing my buffers 
and getting rid of the buffer fsync have completely eliminated by 
IOBOUND issues.

In any case, the way to spot a memory leak is looking at the output of 
top or ps -o args,vsz,rss -u mythtv (or whatever).  Sorting top by 
memory use is a good way to find out where all your RAM is going.  You 
might be surprised to find out that it is something else entirely.