<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Nov 23, 2017 at 7:40 AM, Peter Bennett <span dir="ltr"><<a href="mailto:pb.mythtv@gmail.com" target="_blank">pb.mythtv@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><div><div class="gmail-h5">
<p><br>
</p>
<br>
<div class="gmail-m_1358115403251088994moz-cite-prefix">On 11/23/2017 12:50 AM, Marc Rawji
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, Nov 22, 2017 at 4:41 PM,
Stephen Worthington <span dir="ltr"><<a href="mailto:stephen_agent@jsw.gen.nz" target="_blank">stephen_agent@jsw.gen.nz</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail-m_1358115403251088994gmail-HOEnZb">
<div class="gmail-m_1358115403251088994gmail-h5">On Wed, 22 Nov 2017 10:28:39
-0800, you wrote:<br>
<br>
>On Tue, Nov 21, 2017 at 11:11 PM, Stephen
Worthington <<br>
><a href="mailto:stephen_agent@jsw.gen.nz" target="_blank">stephen_agent@jsw.gen.nz</a>>
wrote:<br>
><br>
>> On Wed, 22 Nov 2017 06:03:31 +0000, you
wrote:<br>
>><br>
>> >Hi Everyone,<br>
>> ><br>
>> >I've been working on an external recorder
for a while. It works most of<br>
>> the<br>
>> >time, but recently i've been getting new
errors.<br>
>> ><br>
>> <snip><br>
>> >My problem below is that I am getting a
"Resource temporarily unavailable<br>
>> >(11)" when mythtv is trying to read from
the stdin (the stdout of my<br>
>> >program with the video stream)<br>
>> ><br>
>> >This seems to happen from this call:<br>
>> ><a href="https://github.com/MythTV/mythtv/blob/master/mythtv/" rel="noreferrer" target="_blank">https://github.com/MythTV/myt<wbr>htv/blob/master/mythtv/</a><br>
>> libs/libmythtv/recorders/Exter<wbr>nalStreamHandler.cpp#L125<br>
>> ><br>
>> >The STDIN stream is configured to be non
blocking, so the errno 11 isn't<br>
>> >unexpected:<br>
>> ><a href="https://github.com/MythTV/mythtv/blob/master/mythtv/" rel="noreferrer" target="_blank">https://github.com/MythTV/myt<wbr>htv/blob/master/mythtv/</a><br>
>> libs/libmythtv/recorders/Exter<wbr>nalStreamHandler.cpp#L343<br>
>> ><br>
>> >I am not sure what to change to keep this
from happening.... it fails<br>
>> about<br>
>> >50% of the time, which is annoying.<br>
>> ><br>
>> >I am looking for advice on a fix, or how
to troubleshoot. I am running<br>
>> 0.29<br>
>> >on Ubuntu 16.04.<br>
>> ><br>
>> >Thanks!<br>
>> >Marc<br>
>> ><br>
>> <snip><br>
>><br>
>> If I am reading the
ExternalStreamHandler.cpp.Exte<wbr>rnIO::Read function<br>
>> correctly, then the only way that can happen
is if your stdout stream<br>
>> indicates that it has data buffered and
available to be read (the<br>
>> poll() call from the Ready() call says there
is data there), but then<br>
>> your stdout fails to provide that data when
it is called from read().<br>
>> The most obvious reason I can think of for
that would be if your code<br>
>> was slow in responding to the read(), due to
some unforeseen<br>
>> circumstance such as garbage collection or
being swapped out at the<br>
>> time. But I would have thought that your
stdout buffer would have<br>
>> been there waiting to be read, rather than
any higher code needing to<br>
>> be run to provide the data. Are you using
buffered I/O on your<br>
>> stdout?<br>
>><br>
>><br>
>Thanks for the quick reply Stephen. I am going to
very quickly expose my<br>
>ignorance on how these FDs work. (just setting
expectations)<br>
><br>
>Yes, I am using buffered output, specifically a
BufferedWriter in Python (<br>
><a href="https://docs.python.org/2/library/io.html#io.BufferedWriter" rel="noreferrer" target="_blank">https://docs.python.org/2/lib<wbr>rary/io.html#io.BufferedWriter</a><wbr>).
I added<br>
>explicit "flush" commands after the 'write'
command, but it didn't help.<br>
><br>
>I'll try dropping the buffered output tonight and
see what happens. I<br>
>didn't think that using a buffered writer would
affect how data is present<br>
>in the stream.<br>
><br>
>Thanks,<br>
>Marc<br>
<br>
</div>
</div>
I would have hoped that using a BufferedWriter would have
provided<br>
more buffering rather than less, but it would be useful to
try a<br>
normal file instead, using the normal buffering, to see if
it makes a<br>
difference. The problem with this situation is that the<br>
ExternalStreamHandler.cpp code needs to be able to read
the data from<br>
the pipe buffer instantly, as it has been told that there
is data<br>
waiting. So if the buffer the data is coming from is not
locked in<br>
RAM, it may not be instantly available. The normal file
I/O buffers<br>
in Python are handled in the underlying C library and
should work the<br>
way ExternalStreamHandler.cpp expects them to, but it is
vaguely<br>
possible that BufferedWriter changes that and accessing
its buffers<br>
requires running the Python interpreter, which could cause
this<br>
problem. The documentation of BufferedWriter is not
specific enough<br>
about how it works to know exactly how its low level
buffering works,<br>
but it is suggestive that the data may be lying in a
higher level<br>
buffer.<br>
<div class="gmail-m_1358115403251088994gmail-HOEnZb">
<div class="gmail-m_1358115403251088994gmail-h5"><br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Just gave it a go and no joy, same behavior as
before...</div>
<div><br>
</div>
<div>In my Python code, the salient part is:</div>
<div>data = fd.read(256)<br>
</div>
<div>
<div>if data:</div>
<div> if (self.last_write_time is None):</div>
<div> self.logger.debug("First bytes written to
stdout")</div>
<div> self.__bin_stdout.write(data)</div>
<div> self.__bin_stdout.flush()</div>
<div> self.last_write_time = time.time()</div>
<div>self.writer_control.wait(0.5)</div>
</div>
<div><br>
</div>
<div>and self.__bin_stdout is equal to sys.stdout.</div>
<div><br>
</div>
<div>Would it have to do with:</div>
<div><a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/poll.html" target="_blank">http://pubs.opengroup.org/<wbr>onlinepubs/9699919799/<wbr>functions/poll.html</a><br>
</div>
<div><br>
</div>
<div>It indicates that for streams, the POLLIN is set even
if the message is of zero length.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Marc</div>
<div> </div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_1358115403251088994mimeAttachmentHeader"></fieldset>
<br>
</blockquote></div></div>
I suggest creating a ticket for this. <br>
<br>
You may want to look at the import recorder. I recently committed a
fix that allows it to be used for recordings. See the fix at
<a class="gmail-m_1358115403251088994moz-txt-link-freetext" href="https://code.mythtv.org/trac/ticket/13114" target="_blank">https://code.mythtv.org/trac/<wbr>ticket/13114</a> and documentation at
<a class="gmail-m_1358115403251088994moz-txt-link-freetext" href="https://www.mythtv.org/wiki/Import_recorder" target="_blank">https://www.mythtv.org/wiki/<wbr>Import_recorder</a> . Currently the fix is
only available in master, scheduled for v30.<span class="gmail-HOEnZb"><font color="#888888"><br>
<br>
Peter</font></span></div></blockquote><div><br></div><div><br></div><div>Thanks for the suggestion, Peter. I'll look at possibly using the import recorder...</div><div><br></div><div>Stephen, I grabbed the latest V30 (master) and hacked up ExternalStreamHandler.cpp with extra log messages. I replaced the Ready function to use select instead of poll and still had the same behavior. I then commented out this line:</div><div><div>- m_error = "Failed to read from External Recorder: " + ENO;</div><div>+ //m_error = "MR: Failed to read from External Recorder: " + ENO;</div></div><div>preventing the hard failure, and got a functioning video stream.</div><div><br></div><div>In the snippet below, FD 62 is the "stdout" with the transport stream. </div><div>The "Checking for data" line comes from the Ready function: 1 indicates there's data to read.</div><div>The empty line in the log is the from the commented out m_error... It's when we get the unexpected "erno 11". </div><div><br></div><div>We can see there's data before and after the error... I am not sure it's a problem with the Ready function, but with the 'read'</div><div><br></div><div><div>2017-11-23 15:07:35.517510 I ExternalRec(/home/mrawji/recorder/record.sh): ProcessCommand('StopStreaming') = 'OK:Stopped'</div><div>2017-11-23 15:07:35.517516 E ExternIO::Read '256' bytes, buffer size 312</div><div>2017-11-23 15:07:35.517690 E MR: Checking for data, return value was: '1', for FD '62'</div><div>2017-11-23 15:07:35.517693 E MR: Checking for data, return value was: '1', for FD '62'</div><div>2017-11-23 15:07:35.517700 E</div><div>2017-11-23 15:07:35.517709 E ExternIO::Read '256' bytes, buffer size 380</div><div>2017-11-23 15:07:35.517822 E MR: Checking for data, return value was: '1', for FD '62'</div><div>2017-11-23 15:07:35.517842 E ExternIO::Read '256' bytes, buffer size 260</div><div>2017-11-23 15:07:35.517879 E MR: Checking for data, return value was: '1', for FD '62'</div><div>2017-11-23 15:07:35.517901 E ExternIO::Read '256' bytes, buffer size 328</div></div><div><br></div><div>I am thinking that I could possibly add a counter on the failed reads and if we hit 2 (or n) in row, then fail... It also looks like it'll read "0 bytes" a few times despite the Ready function returning True. In the trace above, I see 2 0 byte reads, followed by the "erno 11". Then, everything keeps working again (9 microseconds after the error).</div><div><br></div><div>I am writing 256 bytes at a time from my python app... not sure if that means anything.</div><div><br></div><div>Thanks</div><div>Marc</div><div><br></div></div></div></div>