[mythtv-users] why does tv_grab_na_dd/mythfilldatabase use so much memory?

[jds]

hi,
any idea why the whole tv_grab_na_dd/mythfilldatabase operation should
take up so much memory when running?

165MB when loading tv_grab_na_dd data
444MB when running mythfilldatabase

as you can imagine, with "only" 512MB ram, i get to watch the disk LED
a lot while updating the program database.  while that is not an issue
at ~3AM, i thought that someone may be interested to know how big this
operation can get.

the data shown below is after a reboot and mythfrontend/backend were
exited from to free up some RAM.  otherwise, due to swapping, it takes
literally forever to update to db; e.g., my first pass ran >20 hrs and
had not finished when i stopped it.  the box (866Mhz P3) was slogging
due to swap/paging.  as i write this the new pass has just completed,
after about 35 minutes.

ps: this is "built yesterday" xmltv-0.5.33, btw, and the script as is
described in http://www.gossamer-threads.com/lists/mythtv/users/67435
and http://mysettopbox.tv/phpBB2/viewtopic.php?t=1514 (att'd below).

ps
last, and somewhat related question:  what do i set "offset" to now???
i.e. from the script at the bottom of this email,
# replace all data
offset=-1

thanks
jds


INFO: during tv_grab_na_dd loading...
------------------------------------------------------------------
root at mythtv:/home/jds# /usr/bin/mythfilldatabase-dd.sh
using config filename /root/.xmltv/tv_grab_na_dd.conf

**WARNING** Password in config file, protect as required

Fetching from DataDirect
    Fetched 24087 k/bytes in 152 seconds
loading data ...<snip>...

Writing schedule...<snip>...

top - 07:43:02 up 25 min,  3 users,  load average: 1.00, 0.92, 0.54
Tasks:  69 total,   2 running,  67 sleeping,   0 stopped,   0 zombie
Cpu(s):  98.3% user,   1.7% system,   0.0% nice,   0.0% idle
Mem:    515844k total,   407348k used,   108496k free,    10236k buffers
Swap:   627472k total,        0k used,   627472k free,   170492k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command
  493 root      20   0  165m 165m 2224 R 99.3 32.8  19:52.73 tv_grab_na_dd
  517 mythuser  17  10  1000 1000  804 R  0.7  0.2   0:05.66 top
------------------------------------------------------------------


INFO: during mythfilldatabase execution...
------------------------------------------------------------------

Message: Your subscription will expire: 2004-08-07T17:43:45Z
Downloaded 81025 programs in 1142 seconds
### bypassing grabbers, reading directly from file

top - 07:50:53 up 33 min,  3 users,  load average: 1.14, 1.03, 0.73
Tasks:  71 total,   2 running,  69 sleeping,   0 stopped,   0 zombie
Cpu(s):  78.0% user,   4.9% system,   0.0% nice,  17.1% idle
Mem:    515844k total,   512600k used,     3244k free,      652k buffers
Swap:   627472k total,    86980k used,   540492k free,    27380k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command
  544 root      20   0  444m 442m  65m R 71.6 87.9   3:43.21 mythfilldatabas
  543 mythuser  16  10   996  944  800 R  0.7  0.2   0:02.33 top

mythuser at mythtv:~/xmltv $ vmstat 5 10
procs --------memory---------- --swap- -----io------ -system- --cpu--
 r  b   swpd  free buff  cache  si  so  bi   bo   in    cs us sy id wa
 1  0 104024  3272  908  29108  20  31  78  255  159   688 67  3 30  0
 2  0 104024  3252  908  29096   0   0  19  449  207   585 88  3  9  0
 1  0 104024  3892  968  28212  54  16 146  482  219  1345 87  4  9  0
 1  0 104024  3568  976  28456  13   0  34  330  183   740 86  3 11  0
 2  0 104024  3268  984  28648  17   7  58  491  222   672 76  2 21  0
 2  0 104024  3268  976  28536  26 106  44  538  207   563 87  2 11  0
 1  0 104024  3328  960  28364  26  18  37  530  225   562 83  3 14  0
 2  0 104056  3256  960  28280  26   1  88  409  209   490 83  3 14  0
 1  0 104312  3228  952  28144  33   0 106  428  219   887 79  3 18  0
 3  0 104312  3192  944  28072  26   0  58  452  214   523 79  3 18  0

------------------------------------------------------------------



note:
mythuser at mythtv:~/xmltv $ cat /usr/bin/mythfilldatabase-dd.sh
#!/bin/sh
#
# see http://mysettopbox.tv/phpBB2/viewtopic.php?t=1514

# temporary filename
dirname='/tmp'
filename="tv-${RANDOM}${RANDOM}"
xml_file=${dirname}/${filename}.xml

# From step 8
sourceid=8

# replace all data
offset=-1

# How many days worth of guide data to download
numdays=10

/usr/bin/tv_grab_na_dd --output ${xml_file} --days ${numdays} \
  --old-chan-id &&  /usr/bin/mythfilldatabase \
  --file ${sourceid} ${offset} ${xml_file}

# Remove the temporary file
rm ${xml_file}