debugging a new problem

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

debugging a new problem

Rob Janssen
I am debugging a new problem that has first shown after we updated from a fairly-recent master
version to the April 3 version (the one that introduced the new DTMF decoder).

It is happening on a Cubieboard2 where we run 3 remote receivers, two via the built-in soundcard
and a third via:

Bus 002 Device 002: ID 0d8c:000e C-Media Electronics, Inc. Audio Adapter (Planet UP-100, Genius G-Talk)

The config has 3 receivers configured, with TX=NONE. All 3 receivers are running the new DTMF decoder.

The remotetrx program runs OK for a day or so, then crashes.  I have run it interactively and one
time it crashed with Segmentation violation, it was running as user svxlink then and it did not dump core
because of ulimit setting.   Then I started it as root and I have a core file, but it indicates it crashed
with an Abort.
So it was not completely the same problem, maybe some random memory overwrite that can end in
different ways.

The core file has this stack backtrace:

#0  0xb6bb2f96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
#1  0xb6bc0f8a in raise () from /lib/arm-linux-gnueabihf/libc.so.6
#2  0xb6bc3428 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
#3  0xb6bbc01e in __assert_fail () from /lib/arm-linux-gnueabihf/libc.so.6
#4  0xb6f159f8 in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#5  0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#6  0xb6f1603e in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#7  0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#8  0xb6f1e36a in Async::AudioDelayLine::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#9  0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#10 0x000667a0 in Async::AudioStreamStateDetector::writeSamples(float const*, int) ()
#11 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#12 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
#13 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#14 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#15 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#16 0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#17 0xb6f160ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#18 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#19 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#20 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#21 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#22 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#23 0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#24 0xb6f160ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#25 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#26 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
#27 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#28 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#29 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#30 0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#31 0xb6f160ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#32 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#33 0xb6f2008a in Async::AudioFifo::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#34 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#35 0xb6f1cc04 in Async::AudioDevice::putBlocks(short*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#36 0x00000000 in ?? ()

I think it means that it failed writing audio samples and it is unclear to me why it would even do
that, nowhere is a config that specifies a soundcard for output and I don't see one opened in /proc
either.

But anyway, I again hit the issue (like when debugging a problem with the Voter long ago) that
the message output via assert() is lost because it is written to stderr, which svxlink internally
redirects to the log output via a pipe construct, and does not flush when an assert happens.

So I think I'll try again without logfile, to see if I get the message on the console.
Is there a way to retrieve the actual message from the core dump via gdb?

Interesting point is that shortly before the crash, the logfile shows:

Sun Apr 12 12:04:07 2015: PI2NOS: SetMuteState(NONE)
Sun Apr 12 12:04:07 2015: PI2NOS: SetMuteState(NONE)
Sun Apr 12 12:04:12 2015: PI2NOS: SetMuteState(CONTENT)
Sun Apr 12 12:04:13 2015: PI2NOS: SetMuteState(NONE)
Sun Apr 12 12:04:16 2015: PI2NOS: SetMuteState(CONTENT)
Sun Apr 12 12:04:16 2015: *** ERROR: TCP transmit buffer overflow in NetUplink Rx_GOE.
Sun Apr 12 12:04:16 2015: Rx_GOE: Client disconnected: 44.137.28.114:47420
Sun Apr 12 12:04:16 2015: PI3GOE: The squelch is CLOSED (52.6461)
Sun Apr 12 12:04:17 2015: PI3GOE: The squelch is OPEN (53.7936)
Sun Apr 12 12:04:18 2015: PI2NOS: SetMuteState(NONE)
Sun Apr 12 12:04:22 2015: PI2NOS: SetMuteState(CONTENT)
Sun Apr 12 12:04:23 2015: PI2NOS: SetMuteState(NONE)
Sun Apr 12 12:04:28 2015: PI2NOS: SetMuteState(CONTENT)
Sun Apr 12 12:04:29 2015: PI2NOS: SetMuteState(NONE)
Sun Apr 12 12:04:29 2015: PI2NOS: SetMuteState(CONTENT)
Sun Apr 12 12:04:29 2015: PI2NOS: SetMuteState(NONE)
Sun Apr 12 12:04:39 2015: Rx_GOE: Client connected: 44.137.28.114:49867
Sun Apr 12 12:04:39 2015: PI3GOE: SetMuteState(NONE)
(here the program has crashed)

This client reconnect does not happen often, in fact it looks like it has happened just before every crash we had.
So maybe it is related...


Rob

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Tobias Blomberg
On Sunday 12 April 2015 18:40:31 Rob Janssen wrote:

> I am debugging a new problem that has first shown after we updated from a
> fairly-recent master version to the April 3 version (the one that
> introduced the new DTMF decoder).
>
> It is happening on a Cubieboard2 where we run 3 remote receivers, two via
> the built-in soundcard and a third via:
>
> Bus 002 Device 002: ID 0d8c:000e C-Media Electronics, Inc. Audio Adapter
> (Planet UP-100, Genius G-Talk)
>
> The config has 3 receivers configured, with TX=NONE. All 3 receivers are
> running the new DTMF decoder.
>
> The remotetrx program runs OK for a day or so, then crashes.  I have run it
> interactively and one time it crashed with Segmentation violation, it was
> running as user svxlink then and it did not dump core because of ulimit
> setting.   Then I started it as root and I have a core file, but it
> indicates it crashed with an Abort.
> So it was not completely the same problem, maybe some random memory
> overwrite that can end in different ways.
>
> The core file has this stack backtrace:
>
> #0  0xb6bb2f96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> #1  0xb6bc0f8a in raise () from /lib/arm-linux-gnueabihf/libc.so.6
> #2  0xb6bc3428 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
> #3  0xb6bbc01e in __assert_fail () from /lib/arm-linux-gnueabihf/libc.so.6
> #4  0xb6f159f8 in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #5  0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #6  0xb6f1603e in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #7  0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #8  0xb6f1e36a in Async::AudioDelayLine::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #9  0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #10 0x000667a0 in Async::AudioStreamStateDetector::writeSamples(float
> const*, int) () #11 0xb6f159de in
> Async::AudioSource::sinkWriteSamples(float const*, int) () from
> /usr/lib/libasyncaudio.so.1.3
> #12 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
> #13 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #14 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #15 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #16 0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #17 0xb6f160ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #18 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #19 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #20 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #21 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #22 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #23 0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #24 0xb6f160ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #25 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #26 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
> #27 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #28 0xb6f1e020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #29 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #30 0xb6f15ef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #31 0xb6f160ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #32 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #33 0xb6f2008a in Async::AudioFifo::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #34 0xb6f159de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #35 0xb6f1cc04 in Async::AudioDevice::putBlocks(short*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #36 0x00000000 in ?? ()
>
> I think it means that it failed writing audio samples and it is unclear to
> me why it would even do that, nowhere is a config that specifies a
> soundcard for output and I don't see one opened in /proc either.

Unfortunately the audio pipe framework make errors like this harder to
analyze. It's hard to know where in the code it is crashing. One thing that
would help is to compile with debug information enabled. Add

        -DCMAKE_BUILD_TYPE=RelWithDebInfo

to the cmake command line to get both optimization and debug symbols.


> But anyway, I again hit the issue (like when debugging a problem with the
> Voter long ago) that the message output via assert() is lost because it is
> written to stderr, which svxlink internally redirects to the log output via
> a pipe construct, and does not flush when an assert happens.
>
> So I think I'll try again without logfile, to see if I get the message on
> the console. Is there a way to retrieve the actual message from the core
> dump via gdb?

Maybe, but you'd have to dig deep I think...

I usually run my own node in a tmux session. That way I can disconnect from
the system but always have the possibility to access the console printouts.


> Interesting point is that shortly before the crash, the logfile shows:
>
> Sun Apr 12 12:04:07 2015: PI2NOS: SetMuteState(NONE)
> Sun Apr 12 12:04:07 2015: PI2NOS: SetMuteState(NONE)
> Sun Apr 12 12:04:12 2015: PI2NOS: SetMuteState(CONTENT)
> Sun Apr 12 12:04:13 2015: PI2NOS: SetMuteState(NONE)
> Sun Apr 12 12:04:16 2015: PI2NOS: SetMuteState(CONTENT)
> Sun Apr 12 12:04:16 2015: *** ERROR: TCP transmit buffer overflow in
> NetUplink Rx_GOE. Sun Apr 12 12:04:16 2015: Rx_GOE: Client disconnected:
> 44.137.28.114:47420 Sun Apr 12 12:04:16 2015: PI3GOE: The squelch is CLOSED
> (52.6461)
> Sun Apr 12 12:04:17 2015: PI3GOE: The squelch is OPEN (53.7936)
> Sun Apr 12 12:04:18 2015: PI2NOS: SetMuteState(NONE)
> Sun Apr 12 12:04:22 2015: PI2NOS: SetMuteState(CONTENT)
> Sun Apr 12 12:04:23 2015: PI2NOS: SetMuteState(NONE)
> Sun Apr 12 12:04:28 2015: PI2NOS: SetMuteState(CONTENT)
> Sun Apr 12 12:04:29 2015: PI2NOS: SetMuteState(NONE)
> Sun Apr 12 12:04:29 2015: PI2NOS: SetMuteState(CONTENT)
> Sun Apr 12 12:04:29 2015: PI2NOS: SetMuteState(NONE)
> Sun Apr 12 12:04:39 2015: Rx_GOE: Client connected: 44.137.28.114:49867
> Sun Apr 12 12:04:39 2015: PI3GOE: SetMuteState(NONE)
> (here the program has crashed)
>
> This client reconnect does not happen often, in fact it looks like it has
> happened just before every crash we had. So maybe it is related...

Certainly something to look out for.

If you suspect the new DTMF-decoder, try to enable the old one instead by
setting DTMF_DEC_TYPE=DH1DM

You could also try to pinpoint the problem by going back in git history until
you find where the problem was introduced. Have a look at a previous mail I
sent regarding "git bisect". Don't remember in which conversation that was at
the moment.

73's de SM0SVX / Tobias


>
>
> Rob


------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
SM0SVX wrote:

> This client reconnect does not happen often, in fact it looks like it has
> happened just before every crash we had. So maybe it is related...
> Certainly something to look out for.
>
> If you suspect the new DTMF-decoder, try to enable the old one instead by
> setting DTMF_DEC_TYPE=DH1DM
>
> You could also try to pinpoint the problem by going back in git history until
> you find where the problem was introduced. Have a look at a previous mail I
> sent regarding "git bisect". Don't remember in which conversation that was at
> the moment.
>

I don't really suspect the new DTMF decoder, I only mentioned that because it occurred
first after we have switched to that version, from a master version of maybe 2 months earlier.
But it probably has nothing to do with the DTMF decoder, more likely it is related to another
change, or it is a complete coincidence.

The logfiles we still have available don't show those TCP errors, so either they are now
more likely to be triggered or something has changed in the link. This receiver is connected
to the network via a Ubiquity 6cm WiFi link (Nanobridge M5) which shows a very particular
signal level plot over the day, and those errors all appear when the signal is at its lowest
level.  Probably it only fails when the signal is weak AND there is activity on one or more
of the receivers, causing data to be queued up on the link.
Of course this link loss should not result in a crash...

Over 10 days it has now crashed 4 times or so, this makes it a bit difficult to do systematic
debugging like you suggest.   At the moment it is running to my SSH session and it has
not yet failed in that mode.  But no TCP errors either.  We will have to be patient.

Rob

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
Today it crashed again.   I'll try to recompile with debugging symbols later tonight, but for now
I already saw something interesting in this coredump:

Program terminated with signal 11, Segmentation fault.
#0  0xb6e6f030 in Async::AudioEncoderOpus::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
(gdb) bt
#0  0xb6e6f030 in Async::AudioEncoderOpus::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#1  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#2  0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#3  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#4  0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#5  0xb6e6003e in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#6  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#7  0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#8  0xb6e6003e in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#9  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#10 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#11 0xb6e6003e in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#12 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#13 0xb6e6878a in Async::AudioDelayLine::writeRemainingSamples() ()
    from /usr/lib/libasyncaudio.so.1.3
#14 0xb6e5fa16 in Async::AudioSource::sinkFlushSamples() ()
    from /usr/lib/libasyncaudio.so.1.3
#15 0x00066764 in Async::AudioStreamStateDetector::flushSamples() ()
#16 0xb6e5fa16 in Async::AudioSource::sinkFlushSamples() ()
    from /usr/lib/libasyncaudio.so.1.3
#17 0x000611ce in LocalRxBase::setMuteState(Rx::MuteState) ()
#18 0x00060f64 in LocalRxBase::reset() ()
#19 0x00038fd8 in NetUplink::clientDisconnected(Async::TcpConnection*, Async::TcpConnection::DisconnectReason) ()
#20 0x000390e2 in NetUplink::sendMsg(NetTrxMsg::Msg*) ()
#21 0x0003952c in NetUplink::writeEncodedSamples(void const*, int) ()
#22 0x0003c012 in sigc::internal::slot_call2<sigc::bound_mem_functor2<void, NetUplink, void const*, int>, void, void const*, int>::call_it(sigc::internal::slot_rep*, void const* const&, int const&) ()
#23 0xb6e6f0d6 in Async::AudioEncoderOpus::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#24 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#25 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#26 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#27 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#28 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#29 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#30 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#31 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#32 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#33 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#34 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#35 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#36 0xb6e6836a in Async::AudioDelayLine::writeSamples(float const*, int) ()
---Type <return> to continue, or q <return> to quit---
    from /usr/lib/libasyncaudio.so.1.3
#37 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#38 0x000667a0 in Async::AudioStreamStateDetector::writeSamples(float const*, int) ()
#39 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#40 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
#41 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#42 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#43 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#44 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#45 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#46 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#47 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#48 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#49 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#50 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#51 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#52 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#53 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#54 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
#55 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#56 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#57 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#58 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
    from /usr/lib/libasyncaudio.so.1.3
#59 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#60 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#61 0xb6e6a08a in Async::AudioFifo::writeSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#62 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#63 0xb6e66c04 in Async::AudioDevice::putBlocks(short*, int) ()
    from /usr/lib/libasyncaudio.so.1.3
#64 0x00000000 in ?? ()

As you can see it was already several levels deep in calls writing audio samples, when a signal occurred
because of the TCP error (SIGPIPE) and in its handling it again starts flushing samples.  This finally
ends in a SIGSEGV.   Hopefully with debugging option we again get to see the values of the passed
parameters and/or more correct function names (static functions are probably labeled incorrectly now).

However, when handling socket errors using SIGPIPE there of course is always a risk that the signal
comes in at an unfortunate moment and race conditions or re-entrant calls occur to functions that are not
capable of handling them.

It could be better to disable the use of SIGPIPE and instead use the return value from the write/send to
know that it is failing, and handle that situation explicitly at a convenient point in the code.
(pass the flag MSG_NOSIGNAL to send() calls or set the SIGPIPE signal to SIG_IGN,
then be careful to always check the result of reads and writes of the socket)

Rob

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Tobias Blomberg
On Wednesday 15 April 2015 15:52:53 Rob Janssen wrote:

> Today it crashed again.   I'll try to recompile with debugging symbols later
> tonight, but for now I already saw something interesting in this coredump:
>
> Program terminated with signal 11, Segmentation fault.
> #0  0xb6e6f030 in Async::AudioEncoderOpus::writeSamples(float const*, int)
> () from /usr/lib/libasyncaudio.so.1.3
> (gdb) bt
> #0  0xb6e6f030 in Async::AudioEncoderOpus::writeSamples(float const*, int)
> () from /usr/lib/libasyncaudio.so.1.3
> #1  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #2  0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #3  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #4  0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #5  0xb6e6003e in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #6  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #7  0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #8  0xb6e6003e in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #9  0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #10 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #11 0xb6e6003e in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #12 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #13 0xb6e6878a in Async::AudioDelayLine::writeRemainingSamples() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #14 0xb6e5fa16 in Async::AudioSource::sinkFlushSamples() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #15 0x00066764 in Async::AudioStreamStateDetector::flushSamples() ()
> #16 0xb6e5fa16 in Async::AudioSource::sinkFlushSamples() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #17 0x000611ce in LocalRxBase::setMuteState(Rx::MuteState) ()
> #18 0x00060f64 in LocalRxBase::reset() ()
> #19 0x00038fd8 in NetUplink::clientDisconnected(Async::TcpConnection*,
> Async::TcpConnection::DisconnectReason) () #20 0x000390e2 in
> NetUplink::sendMsg(NetTrxMsg::Msg*) ()
> #21 0x0003952c in NetUplink::writeEncodedSamples(void const*, int) ()
> #22 0x0003c012 in sigc::internal::slot_call2<sigc::bound_mem_functor2<void,
> NetUplink, void const*, int>, void, void const*,
> int>::call_it(sigc::internal::slot_rep*, void const* const&, int const&) ()
> #23 0xb6e6f0d6 in Async::AudioEncoderOpus::writeSamples(float const*, int)
> () from /usr/lib/libasyncaudio.so.1.3
> #24 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #25 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #26 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #27 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #28 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #29 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #30 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #31 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #32 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #33 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #34 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #35 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #36 0xb6e6836a in Async::AudioDelayLine::writeSamples(float const*, int) ()
> ---Type <return> to continue, or q <return> to quit---
>     from /usr/lib/libasyncaudio.so.1.3
> #37 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #38 0x000667a0 in Async::AudioStreamStateDetector::writeSamples(float
> const*, int) () #39 0xb6e5f9de in
> Async::AudioSource::sinkWriteSamples(float const*, int) () from
> /usr/lib/libasyncaudio.so.1.3
> #40 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
> #41 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #42 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #43 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #44 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #45 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #46 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #47 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #48 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #49 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #50 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #51 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #52 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #53 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #54 0x0003e944 in Async::AudioValve::writeSamples(float const*, int) ()
> #55 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #56 0xb6e68020 in Async::AudioSplitter::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #57 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #58 0xb6e5fef2 in Async::AudioProcessor::writeFromBuf() ()
>     from /usr/lib/libasyncaudio.so.1.3
> #59 0xb6e600ec in Async::AudioProcessor::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #60 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #61 0xb6e6a08a in Async::AudioFifo::writeSamples(float const*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #62 0xb6e5f9de in Async::AudioSource::sinkWriteSamples(float const*, int) ()
> from /usr/lib/libasyncaudio.so.1.3
> #63 0xb6e66c04 in Async::AudioDevice::putBlocks(short*, int) ()
>     from /usr/lib/libasyncaudio.so.1.3
> #64 0x00000000 in ?? ()

Some dangerous coding in the Opus, Speex and Gsm encoders (the C++ classes)
made this happen. At network disconnect a buffer overflow could occur in the
AudioEncoderXxx::writeSamples functions since it entered again before the
writeEncodedSamples signal had returned. In the Opus case the buf_len variable
was not reset so it wrote past the end of the sample_buf buffer.

Hopefully the latest code in git master will fix this.


> As you can see it was already several levels deep in calls writing audio
> samples, when a signal occurred because of the TCP error (SIGPIPE) and in
> its handling it again starts flushing samples.  This finally ends in a
> SIGSEGV.   Hopefully with debugging option we again get to see the values
> of the passed parameters and/or more correct function names (static
> functions are probably labeled incorrectly now).
>
> However, when handling socket errors using SIGPIPE there of course is always
> a risk that the signal comes in at an unfortunate moment and race
> conditions or re-entrant calls occur to functions that are not capable of
> handling them.
>
> It could be better to disable the use of SIGPIPE and instead use the return
> value from the write/send to know that it is failing, and handle that
> situation explicitly at a convenient point in the code. (pass the flag
> MSG_NOSIGNAL to send() calls or set the SIGPIPE signal to SIG_IGN, then be
> careful to always check the result of reads and writes of the socket)

SIGPIPE is actually set to be ignored at the start of main in remotetrx.cpp so
that should not be able to cause any problems.

73's de SM0SVX / Tobias

>
> Rob


------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
SM0SVX wrote:
> SIGPIPE is actually set to be ignored at the start of main in remotetrx.cpp so
> that should not be able to cause any problems.
>

Ok it must have been another signal then.  Good that you found and fixed something!

I have now written a shellscript to rebuild, install and restart the software on each of the 9
systems we are running it on, and completed the rebuild of the new version.
(just before you merged the eventsource fork, but that does not really matter as it is of course
already installed on our systems)

So now we can see if this fixes something, and also the NULL CODEC mod is now running.
(I needed to get that script finished first because it required a synchrounous rebuild and there
were just too many commands to enter and too many mistakes possible)
Have to wait for a quiet moment on the repeater at the hour mark to know if it is working.

It happens that I have experimented with the RTL stick yesterday. It worked ok via rtl_tcp
after I had brought that and libusb to the most recent version.
(with the distributed libusb in Debian the rtl_tcp crashed each time it was disconnected)

Now I have briefly tried it again but it appears the SIGLEV calibration is wrong now.  Need
to do that again or the squelch won't close anymore.   Anyway, this was just for fun, the
receiver worked but was seriously underperforming compared to the hardware RX.

Rob

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
Rob Janssen wrote:
>
> It happens that I have experimented with the RTL stick yesterday. It worked ok via rtl_tcp
> after I had brought that and libusb to the most recent version.
> (with the distributed libusb in Debian the rtl_tcp crashed each time it was disconnected)
>
> Now I have briefly tried it again but it appears the SIGLEV calibration is wrong now.  Need
> to do that again or the squelch won't close anymore.   Anyway, this was just for fun, the
> receiver worked but was seriously underperforming compared to the hardware RX.
>

I recalibrated it and it is now much more sensitive!
Probably something was not optimal with the gain settings before, I now can open it full
quieting with low power on my portable...
Good job!

Rob

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Tobias Blomberg
In reply to this post by Rob Janssen
On Sunday 19 April 2015 20:38:02 Rob Janssen wrote:
> SM0SVX wrote:
> > SIGPIPE is actually set to be ignored at the start of main in
> > remotetrx.cpp so that should not be able to cause any problems.
>
> Ok it must have been another signal then.  Good that you found and fixed
> something!

Maybe you are confusing UNIX signals with libsigc++ signals?
The signal that is emitted in the stack trace is defined here:

https://github.com/sm0svx/svxlink/blob/master/src/async/audio/AsyncAudioEncoder.h#L166

which is connected here:

https://github.com/sm0svx/svxlink/blob/master/src/svxlink/remotetrx/NetUplink.cpp#L517-L518

to this function:

https://github.com/sm0svx/svxlink/blob/master/src/svxlink/remotetrx/NetUplink.cpp#L673-L686

 

> I have now written a shellscript to rebuild, install and restart the
> software on each of the 9 systems we are running it on, and completed the
> rebuild of the new version. (just before you merged the eventsource fork,
> but that does not really matter as it is of course already installed on our
> systems)
>
> So now we can see if this fixes something, and also the NULL CODEC mod is
> now running. (I needed to get that script finished first because it
> required a synchrounous rebuild and there were just too many commands to
> enter and too many mistakes possible) Have to wait for a quiet moment on
> the repeater at the hour mark to know if it is working.
>
> It happens that I have experimented with the RTL stick yesterday. It worked
> ok via rtl_tcp after I had brought that and libusb to the most recent
> version.
> (with the distributed libusb in Debian the rtl_tcp crashed each time it was
> disconnected)

rtl_tcp seem to be unstable for certain USB stick models. I have two different
models and one of them cause no problems at all but with the other one rtl_tcp
very often crash when the client disconnect/connect.

A pretty recent addition to SvxLink is direct USB connection support in
SvxLink so rtl_tcp is not necessary. Just set TYPE=RtlUsb in the wideband
receiver configuration section. It is rather untested though so there is a high
risk for bugs.


> Now I have briefly tried it again but it appears the SIGLEV calibration is
> wrong now.  Need to do that again or the squelch won't close anymore.  
> Anyway, this was just for fun, the receiver worked but was seriously
> underperforming compared to the hardware RX.

I recently (yesterday) committed some code that lowered the audio level
produced by the Rtl code by 6dB. I did this to get a better default audio
level.

I think this very simple and cheap receiver hardware can be usable for filling
in smaller areas where coverage is missing. It is also usable as a linking
receiver where the signals are always strong so the gain can be lowered on the
USB stick => less risk to saturate the receiver. The performance is actually
better than one might think.

73's de SM0SVX / Tobias


>
> Rob


------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
SM0SVX wrote:

> On Sunday 19 April 2015 20:38:02 Rob Janssen wrote:
>
>> It happens that I have experimented with the RTL stick yesterday. It worked
>> ok via rtl_tcp after I had brought that and libusb to the most recent
>> version.
>> (with the distributed libusb in Debian the rtl_tcp crashed each time it was
>> disconnected)
> rtl_tcp seem to be unstable for certain USB stick models. I have two different
> models and one of them cause no problems at all but with the other one rtl_tcp
> very often crash when the client disconnect/connect.

Ok, I noted that too but I found that updating libusb-1.0 to version 1.0.19 fixed it.
Before I was using 1.0.10, the version that came with the Debian Wheezy distribution.

>
> A pretty recent addition to SvxLink is direct USB connection support in
> SvxLink so rtl_tcp is not necessary. Just set TYPE=RtlUsb in the wideband
> receiver configuration section. It is rather untested though so there is a high
> risk for bugs.

That is what I am using now.  It works very well until now.  I have defined 3 receivers and
connected them to 3 repeaters and we tested with one of us driving around and it works
really well.   This morning there was some tropo and I heard a mobile station working over
a repeater in Belgium, some 140km away from our receiver.

>
> I think this very simple and cheap receiver hardware can be usable for filling
> in smaller areas where coverage is missing. It is also usable as a linking
> receiver where the signals are always strong so the gain can be lowered on the
> USB stick => less risk to saturate the receiver. The performance is actually
> better than one might think.
>

Yes I am very much surprised.  Initially (saturday) it did not perform that well, but since
yesterday (after updating the sw) it works really well.   I think when working via rtl_tcp
it did not correctly set the gain or so.
We have the stick connected to an omni antenna at 220m AGL in a broadcast tower.
Of course this would not work well barefoot, but there is a narrow 70cm bandpass filter
in front of it.  But on the same antenna, two repeaters are transmitting 7.6 MHz higher.

Rob

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
In reply to this post by Rob Janssen
Rob Janssen wrote:

> SM0SVX wrote:
>> SIGPIPE is actually set to be ignored at the start of main in remotetrx.cpp so
>> that should not be able to cause any problems.
>>
>
> Ok it must have been another signal then.  Good that you found and fixed something!
>
> I have now written a shellscript to rebuild, install and restart the software on each of the 9
> systems we are running it on, and completed the rebuild of the new version.
> (just before you merged the eventsource fork, but that does not really matter as it is of course
> already installed on our systems)

Ok this particular problem (remotetrx crashing) has not happened since April 19th when I
installed your fix, but a few minutes ago it crashed again.

Unfortunately no core dump as it was again running in normal daemon mode, but the log
shows this:

Fri May  1 12:20:30 2015: PI2NOS: The squelch is CLOSED (4.33316)
Fri May  1 12:20:30 2015: PI2NOS: SetMuteState(NONE)
Fri May  1 12:20:37 2015: PI2NOS: The squelch is OPEN (15.71)
Fri May  1 12:20:37 2015: PI2NOS: SetMuteState(NONE)
Fri May  1 12:20:37 2015: PI2NOS: The squelch is CLOSED (11.5475)
Fri May  1 12:20:38 2015: PI2NOS: SetMuteState(NONE)
Fri May  1 12:20:44 2015: PI2NOS: The squelch is OPEN (9.06761)
Fri May  1 12:20:44 2015: PI2NOS: The squelch is CLOSED (4.32011)
Fri May  1 12:20:44 2015: PI2NOS: SetMuteState(NONE)
Fri May  1 12:20:59 2015: PI2NOS: The squelch is OPEN (18.573)
Fri May  1 12:20:59 2015: PI2NOS: SetMuteState(NONE)
Fri May  1 12:21:03 2015: Rx_NOS: Client disconnected: 44.137.42.60:50175
Fri May  1 12:21:03 2015: PI2NOS: The squelch is CLOSED (13.535)
Fri May  1 12:21:03 2015: *** ERROR: TCP transmit error in NetUplink Rx_NOS.
Fri May  1 12:21:03 2015: PI2NOS: The squelch is OPEN (12.6388)
Fri May  1 12:21:03 2015: PI2NOS: The squelch is CLOSED (10.533)
Fri May  1 12:21:15 2015: PI2NOS: The squelch is OPEN (11.5099)
Fri May  1 12:21:20 2015: PI2NOS: The squelch is CLOSED (16.8741)
Fri May  1 12:21:23 2015: Rx_NOS: Client connected: 44.137.42.60:50264
Fri May  1 12:21:23 2015: PI2NOS: SetMuteState(NONE)
Fri May  1 12:21:23 2015: Rx_NOS: Using CODEC "OPUS" to encode RX audio
Fri May  1 12:21:23 2015: ------ Opus encoder parameters ------
Fri May  1 12:21:23 2015: Frame size           = 320
Fri May  1 12:21:23 2015: Complexity           = 10
Fri May  1 12:21:23 2015: Bitrate              = 32000
Fri May  1 12:21:23 2015: VBR                  = YES
Fri May  1 12:21:23 2015: Constrained VBR      = YES
Fri May  1 12:21:23 2015: Maximum audio bw     = MEDIUMBAND
Fri May  1 12:21:23 2015: Audio bw             = FULLBAND
Fri May  1 12:21:23 2015: Signal type          = VOICE
Fri May  1 12:21:23 2015: Application type     = AUDIO
Fri May  1 12:21:23 2015: Inband FEC           = NO
Fri May  1 12:21:23 2015: Expected Packet Loss = 0%
Fri May  1 12:21:23 2015: DTX                  = NO
Fri May  1 12:21:23 2015: --------------------------------------
Fri May  1 12:21:24 2015: PI2NOS: SetMuteState(CONTENT)
Fri May  1 12:21:25 2015: PI2NOS: SetMuteState(NONE)

here it crashed, and I restarted it manually

Fri May  1 12:30:18 2015: RemoteTrx v1.1.99.12 (Apr 19 2015) Copyright (C) 2003-2015 Tobias Blomberg / SM0SVX

So after the TCP error it had continued for some time but it still crashed a bit later.
The earlier errors were *** ERROR: TCP transmit buffer overflow but now it is *** ERROR: TCP transmit error
There were no TCP errors logged in the timeframe after April 19th, so it is not certain that something has changed.

Rob

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
Rob Janssen wrote:
>
> So after the TCP error it had continued for some time but it still crashed a bit later.
> The earlier errors were *** ERROR: TCP transmit buffer overflow but now it is *** ERROR: TCP transmit error
> There were no TCP errors logged in the timeframe after April 19th, so it is not certain that something has changed.
>
> Rob

This now happened on another remotetrx as well, which is running the same code.  Probably after a short link
failure as well:

Tue May  5 17:38:50 2015: LocalRx_NOS: The squelch is OPEN (51.9499)
Tue May  5 17:38:50 2015: LocalRx_NOS: SetMuteState(CONTENT)
Tue May  5 17:39:11 2015: LocalRx_NOS: The squelch is CLOSED (82.835)
Tue May  5 17:39:11 2015: LocalRx_NOS: SetMuteState(NONE)
Tue May  5 17:39:13 2015: LocalRx_NOS: The squelch is OPEN (19.7597)
Tue May  5 17:39:13 2015: LocalRx_NOS: SetMuteState(CONTENT)
Tue May  5 17:39:14 2015: LocalRx_NOS: SetMuteState(NONE)
Tue May  5 17:39:14 2015: LocalRx_NOS: SetMuteState(NONE)
Tue May  5 17:39:18 2015: NetUplinkRX_NOS: Client disconnected: 44.137.42.60:38433
Tue May  5 17:39:18 2015: LocalRx_NOS: The squelch is CLOSED (44.5685)
Tue May  5 17:39:18 2015: *** ERROR: TCP transmit error in NetUplink NetUplinkRX_NOS.
Tue May  5 17:39:38 2015: NetUplinkRX_NOS: Client connected: 44.137.42.60:53680
Tue May  5 17:39:38 2015: LocalRx_NOS: SetMuteState(CONTENT)
Tue May  5 17:39:38 2015: LocalRx_NOS: The squelch is OPEN (44.6394)
Tue May  5 17:39:38 2015: NetUplinkRX_NOS: Using CODEC "OPUS" to encode RX audio
Tue May  5 17:39:38 2015: ------ Opus encoder parameters ------
Tue May  5 17:39:38 2015: Frame size           = 320
Tue May  5 17:39:38 2015: Complexity           = 10
Tue May  5 17:39:38 2015: Bitrate              = 32000
Tue May  5 17:39:38 2015: VBR                  = YES
Tue May  5 17:39:38 2015: Constrained VBR      = YES
Tue May  5 17:39:38 2015: Maximum audio bw     = MEDIUMBAND
Tue May  5 17:39:38 2015: Audio bw             = FULLBAND
Tue May  5 17:39:38 2015: Signal type          = VOICE
Tue May  5 17:39:38 2015: Application type     = AUDIO
Tue May  5 17:39:38 2015: Inband FEC           = NO
Tue May  5 17:39:38 2015: Expected Packet Loss = 0%
Tue May  5 17:39:38 2015: DTX                  = NO
Tue May  5 17:39:38 2015: --------------------------------------
Tue May  5 17:39:41 2015: LocalRx_NOS: SetMuteState(NONE)
Tue May  5 17:44:59 2015: RemoteTrx v1.1.99.12 (Apr 19 2015) Copyright (C) 2003-2015 Tobias Blomberg / SM0SVX

Rob

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Tobias Blomberg
On Tuesday 05 May 2015 17:49:18 Rob Janssen wrote:

> Rob Janssen wrote:
> > So after the TCP error it had continued for some time but it still crashed
> > a bit later. The earlier errors were *** ERROR: TCP transmit buffer
> > overflow but now it is *** ERROR: TCP transmit error There were no TCP
> > errors logged in the timeframe after April 19th, so it is not certain
> > that something has changed.
> >
> > Rob
>
> This now happened on another remotetrx as well, which is running the same
> code.  Probably after a short link failure as well:

I've improved the error printout to also contain the error message set by the
operating system.

A disconnection is now also made directly when the error occur.

This will probably not solve the problem but maybe shed some more light on it.

A stack trace would sure be nice. Maybe you can turn on core dumps?

73's de SM0SVX / Tobias


>
> Tue May  5 17:38:50 2015: LocalRx_NOS: The squelch is OPEN (51.9499)
> Tue May  5 17:38:50 2015: LocalRx_NOS: SetMuteState(CONTENT)
> Tue May  5 17:39:11 2015: LocalRx_NOS: The squelch is CLOSED (82.835)
> Tue May  5 17:39:11 2015: LocalRx_NOS: SetMuteState(NONE)
> Tue May  5 17:39:13 2015: LocalRx_NOS: The squelch is OPEN (19.7597)
> Tue May  5 17:39:13 2015: LocalRx_NOS: SetMuteState(CONTENT)
> Tue May  5 17:39:14 2015: LocalRx_NOS: SetMuteState(NONE)
> Tue May  5 17:39:14 2015: LocalRx_NOS: SetMuteState(NONE)
> Tue May  5 17:39:18 2015: NetUplinkRX_NOS: Client disconnected:
> 44.137.42.60:38433 Tue May  5 17:39:18 2015: LocalRx_NOS: The squelch is
> CLOSED (44.5685) Tue May  5 17:39:18 2015: *** ERROR: TCP transmit error in
> NetUplink NetUplinkRX_NOS. Tue May  5 17:39:38 2015: NetUplinkRX_NOS:
> Client connected: 44.137.42.60:53680 Tue May  5 17:39:38 2015: LocalRx_NOS:
> SetMuteState(CONTENT)
> Tue May  5 17:39:38 2015: LocalRx_NOS: The squelch is OPEN (44.6394)
> Tue May  5 17:39:38 2015: NetUplinkRX_NOS: Using CODEC "OPUS" to encode RX
> audio Tue May  5 17:39:38 2015: ------ Opus encoder parameters ------
> Tue May  5 17:39:38 2015: Frame size           = 320
> Tue May  5 17:39:38 2015: Complexity           = 10
> Tue May  5 17:39:38 2015: Bitrate              = 32000
> Tue May  5 17:39:38 2015: VBR                  = YES
> Tue May  5 17:39:38 2015: Constrained VBR      = YES
> Tue May  5 17:39:38 2015: Maximum audio bw     = MEDIUMBAND
> Tue May  5 17:39:38 2015: Audio bw             = FULLBAND
> Tue May  5 17:39:38 2015: Signal type          = VOICE
> Tue May  5 17:39:38 2015: Application type     = AUDIO
> Tue May  5 17:39:38 2015: Inband FEC           = NO
> Tue May  5 17:39:38 2015: Expected Packet Loss = 0%
> Tue May  5 17:39:38 2015: DTX                  = NO
> Tue May  5 17:39:38 2015: --------------------------------------
> Tue May  5 17:39:41 2015: LocalRx_NOS: SetMuteState(NONE)
> Tue May  5 17:44:59 2015: RemoteTrx v1.1.99.12 (Apr 19 2015) Copyright (C)
> 2003-2015 Tobias Blomberg / SM0SVX
>
> Rob


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
SM0SVX wrote:

> On Tuesday 05 May 2015 17:49:18 Rob Janssen wrote:
>> Rob Janssen wrote:
>>> So after the TCP error it had continued for some time but it still crashed
>>> a bit later. The earlier errors were *** ERROR: TCP transmit buffer
>>> overflow but now it is *** ERROR: TCP transmit error There were no TCP
>>> errors logged in the timeframe after April 19th, so it is not certain
>>> that something has changed.
>>>
>>> Rob
>> This now happened on another remotetrx as well, which is running the same
>> code.  Probably after a short link failure as well:
> I've improved the error printout to also contain the error message set by the
> operating system.
>
> A disconnection is now also made directly when the error occur.
>
> This will probably not solve the problem but maybe shed some more light on it.
>
> A stack trace would sure be nice. Maybe you can turn on core dumps?
>

Ok I am recompiling it now and will run it with core dump enabled on a system where it
sometimes fails.

Rob

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
Rob Janssen wrote:
> Ok I am recompiling it now and will run it with core dump enabled on a system where it
> sometimes fails.
>
> Rob

I have tested it on a new receiver we recently deployed and which has a terrible internet connection.
(still to be checked)

Someone was being received on this receiver and probably there was another pause in the link.
Result:

RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(NONE)
NetUplinkRX_NOS: Client disconnected: 44.137.42.60:56414
RxNOS: The squelch is CLOSED (51.0691)
*** ERROR: TCP transmit error in NetUplink "NetUplinkRX_NOS": Resource temporarily unavailable.
Segmentation fault (core dumped)

Core was generated by `/usr/bin/remotetrx --config=/etc/svxlink/remotetrx.conf --pidfile=/var/run/remo'.
Program terminated with signal 11, Segmentation fault.
#0  Async::TcpConnection::disconnect (this=0x0)
     at /home/cubie/svxlink/src/async/core/AsyncTcpConnection.cpp:215
215       delete wr_watch;
(gdb) bt
#0  Async::TcpConnection::disconnect (this=0x0)
     at /home/cubie/svxlink/src/async/core/AsyncTcpConnection.cpp:215
#1  0x0003c1e8 in NetUplink::sendMsg (this=0xf317f8, msg=0xf78d90)
     at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:622
#2  0x0003c678 in NetUplink::writeEncodedSamples (this=0xf317f8,
     buf=<optimized out>, size=<optimized out>)
     at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:687
#3  0x0003f16a in operator() (this=<optimized out>, _A_a1=<optimized out>,
     _A_a2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1917
#4  operator()<const void* const&, const int&> (this=<optimized out>,
     _A_arg1=<optimized out>, _A_arg2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:103
#5  sigc::internal::slot_call2<sigc::bound_mem_functor2<void, NetUplink, void const*, int>, void, void const*, int>::call_it (rep=<optimized out>,
     a_1=<optimized out>, a_2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/slot.h:173
#6  0xb6e38fb6 in emit (_A_a2=@0xbe8a8924: 71, _A_a1=@0xbe8a8928: 0xbe8a7974,
     impl=0xf66288) at /usr/include/sigc++-2.0/sigc++/signal.h:1252
#7  emit (_A_a2=@0xbe8a8924: 71, _A_a1=@0xbe8a8928: 0xbe8a7974, this=0xf669d4)
     at /usr/include/sigc++-2.0/sigc++/signal.h:2891
#8  operator() (_A_a2=@0xbe8a8924: 71, _A_a1=@0xbe8a8928: 0xbe8a7974,
     this=0xf669d4) at /usr/include/sigc++-2.0/sigc++/signal.h:2899
#9  Async::AudioEncoderOpus::writeSamples (this=0xf669c0,
     samples=<optimized out>, count=85)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioEncoderOpus.cpp:646
#10 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#11 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf2db3c, this=0xf66fd8)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#12 writeSamples (len=85, samples=0xf2db3c, this=0xf32c90)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#13 Async::AudioSplitter::writeSamples (this=0xf32c90, samples=0xf2db3c,
     len=85) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#14 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#15 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#16 Async::AudioProcessor::writeFromBuf (this=0xf2db18)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#17 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf2db18,
     samples=0xf2d848, len=0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#18 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#19 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#20 Async::AudioProcessor::writeFromBuf (this=0xf2d6d0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#21 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf2d6d0,
     samples=0xf2d3a0, len=0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#22 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#23 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#24 Async::AudioProcessor::writeFromBuf (this=0xf2d228)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#25 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf2d228,
     samples=0xbe8a8bbc, len=0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#26 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#27 0xb6e322d2 in Async::AudioDelayLine::writeSamples (this=0xf2bc80,
     samples=0xf29ba4, count=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDelayLine.cpp:240
#28 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#29 0x000628ac in writeSamples (count=85, samples=0xf29ba4, this=0xf2ba58)
     at /home/cubie/svxlink/src/build/include/AsyncAudioPassthrough.h:143
#30 Async::AudioStreamStateDetector::writeSamples (this=0xf2ba58,
     samples=0xf29ba4, count=85)
     at /home/cubie/svxlink/src/build/include/AsyncAudioStreamStateDetector.h:142
#31 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#32 0x00041954 in Async::AudioValve::writeSamples (this=0xf2ba90,
     samples=<optimized out>, count=<optimized out>)
     at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
#33 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#34 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf29ba4, this=0xf29b58)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#35 writeSamples (len=85, samples=0xf29ba4, this=0xf2a090)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#36 Async::AudioSplitter::writeSamples (this=0xf2a090, samples=0xf29ba4,
     len=85) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#37 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#38 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#39 Async::AudioProcessor::writeFromBuf (this=0xf29b80)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#40 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf29b80,
     samples=0xf29518, len=0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#41 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#42 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf293c4, this=0xf29b30)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#43 writeSamples (len=85, samples=0xf293c4, this=0xf29ae0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#44 Async::AudioSplitter::writeSamples (this=0xf29ae0, samples=0xf293c4,
     len=85) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#45 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#46 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf293c4, this=0xf29850)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#47 writeSamples (len=85, samples=0xf293c4, this=0xf29800)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#48 Async::AudioSplitter::writeSamples (this=0xf29800, samples=0xf293c4,
     len=85) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#49 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#50 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#51 Async::AudioProcessor::writeFromBuf (this=0xf293a0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#52 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf293a0,
     samples=0xf1e888, len=0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#53 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#54 0x00041954 in Async::AudioValve::writeSamples (this=0xf28b08,
     samples=<optimized out>, count=<optimized out>)
     at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
#55 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#56 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf1e734, this=0xf29368)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#57 writeSamples (len=85, samples=0xf1e734, this=0xf28a38)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#58 Async::AudioSplitter::writeSamples (this=0xf28a38, samples=0xf1e734,
     len=85) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#59 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#60 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#61 Async::AudioProcessor::writeFromBuf (this=0xf1e710)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#62 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf1e710,
     samples=0xf2816c, len=1)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#63 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#64 0xb6e33c4c in writeSamplesFromFifo (this=0xf278e0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:380
#65 Async::AudioFifo::writeSamplesFromFifo (this=0xf278e0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:365
#66 0xb6e33d8c in writeSamples (count=1024, samples=0xbe8a8e28, this=0xf278e0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:279
#67 Async::AudioFifo::writeSamples (this=0xf278e0, samples=0xbe8a8e28,
     count=1024) at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:225
#68 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#69 0xb6e30718 in Async::AudioDevice::putBlocks (this=0xf1e668,
     buf=<optimized out>, frame_cnt=1024)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDevice.cpp:282
#70 0xb6e3b74c in Async::AudioDeviceAlsa::audioReadHandler (this=0xf1e668,
     watch=0xf2e0f0, revents=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:412
#71 0xb6e3bcda in operator() (this=<optimized out>, _A_a1=<optimized out>,
     _A_a2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1917
#72 operator()<Async::FdWatch* const&, const short unsigned int&> (
     this=<optimized out>, _A_arg1=<optimized out>, _A_arg2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:103
#73 sigc::internal::slot_call2<sigc::bound_mem_functor2<void, Async::AudioDeviceAlsa, Async::FdWatch*, unsigned short>, void, Async::FdWatch*, unsigned short>::call_it (rep=<optimized out>, a_1=<optimized out>, a_2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/slot.h:173
#74 0xb6e3c4fc in emit (_A_a2=@0xbe8aaece: 1, _A_a1=@0xbe8aaea4: 0xf2e0f0,
     impl=0xf2df70) at /usr/include/sigc++-2.0/sigc++/signal.h:1252
#75 emit (_A_a2=@0xbe8aaece: 1, _A_a1=@0xbe8aaea4: 0xf2e0f0,
     this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2891
#76 operator() (_A_a2=@0xbe8aaece: 1, _A_a1=@0xbe8aaea4: 0xf2e0f0,
     this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2899
#77 Async::AudioDeviceAlsa::AlsaWatch::readEvent (this=<optimized out>,
     watch=0xf2e0f0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:154
#78 0xb6e3bcb4 in operator() (this=<optimized out>, _A_a1=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1851
#79 operator()<Async::FdWatch* const&> (this=<optimized out>,
     _A_arg1=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:84
#80 sigc::internal::slot_call1<sigc::bound_mem_functor1<void, Async::AudioDeviceAlsa::AlsaWatch, Async::FdWatch*>, void, Async::FdWatch*>::call_it (
     rep=<optimized out>, a_1=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/slot.h:137
#81 0xb6e52038 in emit (_A_a1=@0xf2e124: 0xf2e0f0, impl=0xf2e728)
     at /usr/include/sigc++-2.0/sigc++/signal.h:1010
#82 emit (_A_a1=@0xf2e124: 0xf2e0f0, this=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/signal.h:2781
#83 operator() (_A_a1=@0xf2e124: 0xf2e0f0, this=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/signal.h:2789
#84 Async::CppApplication::exec (this=0xbe8ab140)
     at /home/cubie/svxlink/src/async/cpp/AsyncCppApplication.cpp:277
#85 0x00039f9c in main (argc=<optimized out>, argv=<optimized out>)
     at /home/cubie/svxlink/src/svxlink/remotetrx/remotetrx.cpp:558
(gdb)

At an earlier attempt after the connection was stuck I hit Ctrl-C and got:
*** error reading from stdin
remotetrx: /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168: int Async::AudioSource::sinkWriteSamples(const float*, int): Assertion `len > 0' failed.
Aborted (core dumped)

(gdb) bt
#0  0xb6b82f96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
#1  0xb6b90f8a in raise () from /lib/arm-linux-gnueabihf/libc.so.6
#2  0xb6b93428 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
#3  0xb6b8c01e in __assert_fail () from /lib/arm-linux-gnueabihf/libc.so.6
#4  0xb6ee4710 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168
#5  0xb6ee4c0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#6  Async::AudioProcessor::writeFromBuf (this=0x1abdb18)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#7  0xb6ee47e8 in Async::AudioSource::unregisterSinkInternal (this=0x1abdb28,
     is_being_destroyed=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:327
#8  0xb6ee47da in Async::AudioSource::unregisterSinkInternal (this=0x1aadc88,
     is_being_destroyed=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:316
#9  0xb6ee4918 in Async::AudioSink::unregisterSource (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSink.cpp:139
#10 0xb6ee4970 in Async::AudioSink::~AudioSink (this=0x1ac2c90,
     __in_chrg=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSink.cpp:116
#11 0xb6eebd48 in Async::AudioSplitter::~AudioSplitter (this=0x1ac2c90,
     __in_chrg=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:248
#12 0xb6eebd98 in Async::AudioSplitter::~AudioSplitter (this=0x1ac2c90,
     __in_chrg=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:248
#13 0x0003bc94 in NetUplink::~NetUplink (this=0x1ac17f8,
     __in_chrg=<optimized out>)
     at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:151
#14 0x0003bdb0 in NetUplink::~NetUplink (this=0x1ac17f8,
     __in_chrg=<optimized out>)
     at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:157
#15 0x0003b500 in TrxHandler::~TrxHandler (this=0x1aadb58,
     __in_chrg=<optimized out>)
     at /home/cubie/svxlink/src/svxlink/remotetrx/TrxHandler.cpp:127
#16 0x00039fb6 in main (argc=<optimized out>, argv=<optimized out>)
     at /home/cubie/svxlink/src/svxlink/remotetrx/remotetrx.cpp:569


Rob

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Tobias Blomberg
On Sunday 10 May 2015 23:34:04 Rob Janssen wrote:

> Rob Janssen wrote:
> > Ok I am recompiling it now and will run it with core dump enabled on a
> > system where it sometimes fails.
> >
> > Rob
>
> I have tested it on a new receiver we recently deployed and which has a
> terrible internet connection. (still to be checked)
>
> Someone was being received on this receiver and probably there was another
> pause in the link. Result:
>
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(NONE)
> NetUplinkRX_NOS: Client disconnected: 44.137.42.60:56414
> RxNOS: The squelch is CLOSED (51.0691)
> *** ERROR: TCP transmit error in NetUplink "NetUplinkRX_NOS": Resource
> temporarily unavailable. Segmentation fault (core dumped)
>
> Core was generated by `/usr/bin/remotetrx
> --config=/etc/svxlink/remotetrx.conf --pidfile=/var/run/remo'. Program
> terminated with signal 11, Segmentation fault.
> #0  Async::TcpConnection::disconnect (this=0x0)
>      at /home/cubie/svxlink/src/async/core/AsyncTcpConnection.cpp:215
> 215       delete wr_watch;

Ooops, sorry. That was due to my last bugfix. Instead of fixing it I added a new
bug. New code in master now.

The stack trace look a little bit strange, probably due to optimizations. Is
it possible for you to run the software without optimizations turned on or
will it be too slow then?

The second issue I need to look at further. It probably only (mostly?) happen
when quitting the application so not as serious as the first one.

73's de SM0SVX / Tobias


> (gdb) bt
> #0  Async::TcpConnection::disconnect (this=0x0)
>      at /home/cubie/svxlink/src/async/core/AsyncTcpConnection.cpp:215
> #1  0x0003c1e8 in NetUplink::sendMsg (this=0xf317f8, msg=0xf78d90)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:622
> #2  0x0003c678 in NetUplink::writeEncodedSamples (this=0xf317f8,
>      buf=<optimized out>, size=<optimized out>)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:687
> #3  0x0003f16a in operator() (this=<optimized out>, _A_a1=<optimized out>,
>      _A_a2=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1917
> #4  operator()<const void* const&, const int&> (this=<optimized out>,
>      _A_arg1=<optimized out>, _A_arg2=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:103
> #5  sigc::internal::slot_call2<sigc::bound_mem_functor2<void, NetUplink,
> void const*, int>, void, void const*, int>::call_it (rep=<optimized out>,
> a_1=<optimized out>, a_2=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/slot.h:173
> #6  0xb6e38fb6 in emit (_A_a2=@0xbe8a8924: 71, _A_a1=@0xbe8a8928:
> 0xbe8a7974, impl=0xf66288) at /usr/include/sigc++-2.0/sigc++/signal.h:1252
> #7  emit (_A_a2=@0xbe8a8924: 71, _A_a1=@0xbe8a8928: 0xbe8a7974,
> this=0xf669d4) at /usr/include/sigc++-2.0/sigc++/signal.h:2891
> #8  operator() (_A_a2=@0xbe8a8924: 71, _A_a1=@0xbe8a8928: 0xbe8a7974,
>      this=0xf669d4) at /usr/include/sigc++-2.0/sigc++/signal.h:2899
> #9  Async::AudioEncoderOpus::writeSamples (this=0xf669c0,
>      samples=<optimized out>, count=85)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioEncoderOpus.cpp:646
> #10 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #11 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf2db3c, this=0xf66fd8)
> at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #12
> writeSamples (len=85, samples=0xf2db3c, this=0xf32c90)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #13 Async::AudioSplitter::writeSamples (this=0xf32c90, samples=0xf2db3c,
>      len=85) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #14
> 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #15 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #16 Async::AudioProcessor::writeFromBuf (this=0xf2db18)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #17 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf2db18,
>      samples=0xf2d848, len=0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #18 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #19 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #20 Async::AudioProcessor::writeFromBuf (this=0xf2d6d0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #21 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf2d6d0,
>      samples=0xf2d3a0, len=0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #22 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #23 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #24 Async::AudioProcessor::writeFromBuf (this=0xf2d228)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #25 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf2d228,
>      samples=0xbe8a8bbc, len=0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #26 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #27 0xb6e322d2 in Async::AudioDelayLine::writeSamples (this=0xf2bc80,
>      samples=0xf29ba4, count=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDelayLine.cpp:240
> #28 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #29 0x000628ac in writeSamples (count=85, samples=0xf29ba4, this=0xf2ba58)
>      at /home/cubie/svxlink/src/build/include/AsyncAudioPassthrough.h:143
> #30 Async::AudioStreamStateDetector::writeSamples (this=0xf2ba58,
>      samples=0xf29ba4, count=85)
>      at
> /home/cubie/svxlink/src/build/include/AsyncAudioStreamStateDetector.h:142
> #31 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #32 0x00041954 in Async::AudioValve::writeSamples (this=0xf2ba90,
>      samples=<optimized out>, count=<optimized out>)
>      at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
> #33 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #34 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf29ba4, this=0xf29b58)
> at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #35
> writeSamples (len=85, samples=0xf29ba4, this=0xf2a090)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #36 Async::AudioSplitter::writeSamples (this=0xf2a090, samples=0xf29ba4,
>      len=85) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #37
> 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #38 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #39 Async::AudioProcessor::writeFromBuf (this=0xf29b80)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #40 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf29b80,
>      samples=0xf29518, len=0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #41 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #42 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf293c4, this=0xf29b30)
> at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #43
> writeSamples (len=85, samples=0xf293c4, this=0xf29ae0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #44 Async::AudioSplitter::writeSamples (this=0xf29ae0, samples=0xf293c4,
>      len=85) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #45
> 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #46 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf293c4, this=0xf29850)
> at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #47
> writeSamples (len=85, samples=0xf293c4, this=0xf29800)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #48 Async::AudioSplitter::writeSamples (this=0xf29800, samples=0xf293c4,
>      len=85) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #49
> 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #50 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #51 Async::AudioProcessor::writeFromBuf (this=0xf293a0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #52 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf293a0,
>      samples=0xf1e888, len=0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #53 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #54 0x00041954 in Async::AudioValve::writeSamples (this=0xf28b08,
>      samples=<optimized out>, count=<optimized out>)
>      at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
> #55 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #56 0xb6e31f84 in sinkWriteSamples (len=85, samples=0xf1e734, this=0xf29368)
> at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #57
> writeSamples (len=85, samples=0xf1e734, this=0xf28a38)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #58 Async::AudioSplitter::writeSamples (this=0xf28a38, samples=0xf1e734,
>      len=85) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #59
> 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #60 0xb6e2ac0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #61 Async::AudioProcessor::writeFromBuf (this=0xf1e710)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #62 0xb6e2ae04 in Async::AudioProcessor::writeSamples (this=0xf1e710,
>      samples=0xf2816c, len=1)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #63 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #64 0xb6e33c4c in writeSamplesFromFifo (this=0xf278e0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:380
> #65 Async::AudioFifo::writeSamplesFromFifo (this=0xf278e0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:365
> #66 0xb6e33d8c in writeSamples (count=1024, samples=0xbe8a8e28,
> this=0xf278e0) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:279 #67
> Async::AudioFifo::writeSamples (this=0xf278e0, samples=0xbe8a8e28,
> count=1024) at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:225
> #68 0xb6e2a6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #69 0xb6e30718 in Async::AudioDevice::putBlocks (this=0xf1e668,
>      buf=<optimized out>, frame_cnt=1024)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDevice.cpp:282
> #70 0xb6e3b74c in Async::AudioDeviceAlsa::audioReadHandler (this=0xf1e668,
>      watch=0xf2e0f0, revents=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:412
> #71 0xb6e3bcda in operator() (this=<optimized out>, _A_a1=<optimized out>,
>      _A_a2=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1917
> #72 operator()<Async::FdWatch* const&, const short unsigned int&> (
>      this=<optimized out>, _A_arg1=<optimized out>, _A_arg2=<optimized out>)
> at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:103 #73
> sigc::internal::slot_call2<sigc::bound_mem_functor2<void,
> Async::AudioDeviceAlsa, Async::FdWatch*, unsigned short>, void,
> Async::FdWatch*, unsigned short>::call_it (rep=<optimized out>,
> a_1=<optimized out>, a_2=<optimized out>) at
> /usr/include/sigc++-2.0/sigc++/functors/slot.h:173
> #74 0xb6e3c4fc in emit (_A_a2=@0xbe8aaece: 1, _A_a1=@0xbe8aaea4: 0xf2e0f0,
>      impl=0xf2df70) at /usr/include/sigc++-2.0/sigc++/signal.h:1252
> #75 emit (_A_a2=@0xbe8aaece: 1, _A_a1=@0xbe8aaea4: 0xf2e0f0,
>      this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2891
> #76 operator() (_A_a2=@0xbe8aaece: 1, _A_a1=@0xbe8aaea4: 0xf2e0f0,
>      this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2899
> #77 Async::AudioDeviceAlsa::AlsaWatch::readEvent (this=<optimized out>,
>      watch=0xf2e0f0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:154
> #78 0xb6e3bcb4 in operator() (this=<optimized out>, _A_a1=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1851
> #79 operator()<Async::FdWatch* const&> (this=<optimized out>,
>      _A_arg1=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:84
> #80 sigc::internal::slot_call1<sigc::bound_mem_functor1<void,
> Async::AudioDeviceAlsa::AlsaWatch, Async::FdWatch*>, void,
> Async::FdWatch*>::call_it ( rep=<optimized out>, a_1=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/slot.h:137
> #81 0xb6e52038 in emit (_A_a1=@0xf2e124: 0xf2e0f0, impl=0xf2e728)
>      at /usr/include/sigc++-2.0/sigc++/signal.h:1010
> #82 emit (_A_a1=@0xf2e124: 0xf2e0f0, this=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/signal.h:2781
> #83 operator() (_A_a1=@0xf2e124: 0xf2e0f0, this=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/signal.h:2789
> #84 Async::CppApplication::exec (this=0xbe8ab140)
>      at /home/cubie/svxlink/src/async/cpp/AsyncCppApplication.cpp:277
> #85 0x00039f9c in main (argc=<optimized out>, argv=<optimized out>)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/remotetrx.cpp:558
> (gdb)
>
> At an earlier attempt after the connection was stuck I hit Ctrl-C and got:
> *** error reading from stdin
> remotetrx: /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168: int
> Async::AudioSource::sinkWriteSamples(const float*, int): Assertion `len >
> 0' failed. Aborted (core dumped)
>
> (gdb) bt
> #0  0xb6b82f96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> #1  0xb6b90f8a in raise () from /lib/arm-linux-gnueabihf/libc.so.6
> #2  0xb6b93428 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
> #3  0xb6b8c01e in __assert_fail () from /lib/arm-linux-gnueabihf/libc.so.6
> #4  0xb6ee4710 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168
> #5  0xb6ee4c0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #6  Async::AudioProcessor::writeFromBuf (this=0x1abdb18)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #7  0xb6ee47e8 in Async::AudioSource::unregisterSinkInternal
> (this=0x1abdb28, is_being_destroyed=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:327
> #8  0xb6ee47da in Async::AudioSource::unregisterSinkInternal
> (this=0x1aadc88, is_being_destroyed=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:316
> #9  0xb6ee4918 in Async::AudioSink::unregisterSource (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSink.cpp:139
> #10 0xb6ee4970 in Async::AudioSink::~AudioSink (this=0x1ac2c90,
>      __in_chrg=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSink.cpp:116
> #11 0xb6eebd48 in Async::AudioSplitter::~AudioSplitter (this=0x1ac2c90,
>      __in_chrg=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:248
> #12 0xb6eebd98 in Async::AudioSplitter::~AudioSplitter (this=0x1ac2c90,
>      __in_chrg=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:248
> #13 0x0003bc94 in NetUplink::~NetUplink (this=0x1ac17f8,
>      __in_chrg=<optimized out>)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:151
> #14 0x0003bdb0 in NetUplink::~NetUplink (this=0x1ac17f8,
>      __in_chrg=<optimized out>)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/NetUplink.cpp:157
> #15 0x0003b500 in TrxHandler::~TrxHandler (this=0x1aadb58,
>      __in_chrg=<optimized out>)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/TrxHandler.cpp:127
> #16 0x00039fb6 in main (argc=<optimized out>, argv=<optimized out>)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/remotetrx.cpp:569
>
>
> Rob


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
SM0SVX wrote:

> On Sunday 10 May 2015 23:34:04 Rob Janssen wrote:
>> Rob Janssen wrote:
>>> Ok I am recompiling it now and will run it with core dump enabled on a
>>> system where it sometimes fails.
>>>
>>> Rob
>> I have tested it on a new receiver we recently deployed and which has a
>> terrible internet connection. (still to be checked)
>>
>> Someone was being received on this receiver and probably there was another
>> pause in the link. Result:
>>
>> RxNOS: SetMuteState(CONTENT)
>> RxNOS: SetMuteState(NONE)
>> RxNOS: SetMuteState(NONE)
>> NetUplinkRX_NOS: Client disconnected: 44.137.42.60:56414
>> RxNOS: The squelch is CLOSED (51.0691)
>> *** ERROR: TCP transmit error in NetUplink "NetUplinkRX_NOS": Resource
>> temporarily unavailable. Segmentation fault (core dumped)
>>
>> Core was generated by `/usr/bin/remotetrx
>> --config=/etc/svxlink/remotetrx.conf --pidfile=/var/run/remo'. Program
>> terminated with signal 11, Segmentation fault.
>> #0  Async::TcpConnection::disconnect (this=0x0)
>>       at /home/cubie/svxlink/src/async/core/AsyncTcpConnection.cpp:215
>> 215       delete wr_watch;
> Ooops, sorry. That was due to my last bugfix. Instead of fixing it I added a new
> bug. New code in master now.
>
> The stack trace look a little bit strange, probably due to optimizations. Is
> it possible for you to run the software without optimizations turned on or
> will it be too slow then?
>
> The second issue I need to look at further. It probably only (mostly?) happen
> when quitting the application so not as serious as the first one.
>
> 73's de SM0SVX / Tobias

I had started the new master code interactive now (but still a RelWithDebugInfo compile) and it crashes
in the same way as I mentioned after the ^C but this time a while after the connection had failed
and was re-established:

RxNOS: The squelch is OPEN (28.3958)
RxNOS: SetMuteState(NONE)
RxNOS: The squelch is CLOSED (21.5164)
RxNOS: The squelch is OPEN (10.3608)
*** ERROR: TCP transmit buffer overflow in NetUplink NetUplinkRX_NOS.
NetUplinkRX_NOS: Client disconnected: 44.137.42.60:55259
RxNOS: The squelch is CLOSED (21.5961)
RxNOS: The squelch is OPEN (21.7056)
RxNOS: The squelch is CLOSED (26.2238)
NetUplinkRX_UTR: Client disconnected: 44.137.42.10:35580
NetUplinkRX_UTR: Client connected: 44.137.42.10:36362
RxUTR: SetMuteState(NONE)
NetUplinkRX_UTR: Using CODEC "OPUS" to encode RX audio
------ Opus encoder parameters ------
Frame size           = 320
Complexity           = 10
Bitrate              = 32000
VBR                  = YES
Constrained VBR      = YES
Maximum audio bw     = MEDIUMBAND
Audio bw             = FULLBAND
Signal type          = VOICE
Application type     = AUDIO
Inband FEC           = NO
Expected Packet Loss = 0%
DTX                  = NO
--------------------------------------
NetUplinkRX_NOS: Client connected: 44.137.42.60:57534
RxNOS: SetMuteState(NONE)
NetUplinkRX_NOS: Using CODEC "OPUS" to encode RX audio
------ Opus encoder parameters ------
Frame size           = 320
Complexity           = 10
Bitrate              = 32000
VBR                  = YES
Constrained VBR      = YES
Maximum audio bw     = MEDIUMBAND
Audio bw             = FULLBAND
Signal type          = VOICE
Application type     = AUDIO
Inband FEC           = NO
Expected Packet Loss = 0%
DTX                  = NO
--------------------------------------
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: SetMuteState(CONTENT)
RxNOS: SetMuteState(NONE)
RxNOS: The squelch is OPEN (19.9524)
remotetrx: /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168: int Async::AudioSource::sinkWriteSamples(const float*, int): Assertion `len > 0' failed.
Aborted (core dumped)

This may be the same scenario as I have posted before, usually we run it as a daemon and with logfile, so no Assert failed/Aborted messages and no core dumps.
It would be nice when the error logging construct is somehow modified to pass assertion messages to the logfile...

The stack backtrace is again very deep:

Program terminated with signal 6, Aborted.
#0  0xb6afaf96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
(gdb) bt
#0  0xb6afaf96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
#1  0xb6b08f8a in raise () from /lib/arm-linux-gnueabihf/libc.so.6
#2  0xb6b0b428 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
#3  0xb6b0401e in __assert_fail () from /lib/arm-linux-gnueabihf/libc.so.6
#4  0xb6e5c710 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168
#5  0xb6e5cc0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#6  Async::AudioProcessor::writeFromBuf (this=0x1255228)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#7  0xb6e5cd56 in Async::AudioProcessor::writeSamples (this=0x1255228,
     samples=0xbed3bab8, len=65)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:144
#8  0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#9  0xb6e642d2 in Async::AudioDelayLine::writeSamples (this=0x1253c80,
     samples=0x1251ba4, count=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDelayLine.cpp:240
#10 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#11 0x000628b4 in writeSamples (count=65, samples=0x1251ba4, this=0x1253a58)
     at /home/cubie/svxlink/src/build/include/AsyncAudioPassthrough.h:143
#12 Async::AudioStreamStateDetector::writeSamples (this=0x1253a58,
     samples=0x1251ba4, count=65)
     at /home/cubie/svxlink/src/build/include/AsyncAudioStreamStateDetector.h:142
#13 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#14 0x00041958 in Async::AudioValve::writeSamples (this=0x1253a90,
     samples=<optimized out>, count=<optimized out>)
     at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
#15 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#16 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x1251ba4, this=0x1251b58)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#17 writeSamples (len=65, samples=0x1251ba4, this=0x1252090)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#18 Async::AudioSplitter::writeSamples (this=0x1252090, samples=0x1251ba4,
     len=65) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#19 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#20 0xb6e5cc0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#21 Async::AudioProcessor::writeFromBuf (this=0x1251b80)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#22 0xb6e5ce04 in Async::AudioProcessor::writeSamples (this=0x1251b80,
     samples=0x12514c8, len=0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#23 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#24 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x12513c4, this=0x1251b30)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#25 writeSamples (len=65, samples=0x12513c4, this=0x1251ae0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#26 Async::AudioSplitter::writeSamples (this=0x1251ae0, samples=0x12513c4,
     len=65) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#27 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#28 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x12513c4, this=0x1251850)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#29 writeSamples (len=65, samples=0x12513c4, this=0x1251800)
---Type <return> to continue, or q <return> to quit---
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#30 Async::AudioSplitter::writeSamples (this=0x1251800, samples=0x12513c4,
     len=65) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#31 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#32 0xb6e5cc0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#33 Async::AudioProcessor::writeFromBuf (this=0x12513a0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#34 0xb6e5ce04 in Async::AudioProcessor::writeSamples (this=0x12513a0,
     samples=0x1246838, len=0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#35 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#36 0x00041958 in Async::AudioValve::writeSamples (this=0x1250b08,
     samples=<optimized out>, count=<optimized out>)
     at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
#37 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#38 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x1246734, this=0x1251368)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142
#39 writeSamples (len=65, samples=0x1246734, this=0x1250a38)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
#40 Async::AudioSplitter::writeSamples (this=0x1250a38, samples=0x1246734,
     len=65) at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329
#41 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#42 0xb6e5cc0a in writeFromBuf (this=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
#43 Async::AudioProcessor::writeFromBuf (this=0x1246710)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
#44 0xb6e5ce04 in Async::AudioProcessor::writeSamples (this=0x1246710,
     samples=0x1250948, len=2)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
#45 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#46 0xb6e65c4c in writeSamplesFromFifo (this=0x124f8e0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:380
#47 Async::AudioFifo::writeSamplesFromFifo (this=0x124f8e0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:365
#48 0xb6e65d8c in writeSamples (count=1024, samples=0xbed3be28, this=0x124f8e0)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:279
#49 Async::AudioFifo::writeSamples (this=0x124f8e0, samples=0xbed3be28,
     count=1024) at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:225
#50 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
     samples=<optimized out>, len=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
#51 0xb6e62718 in Async::AudioDevice::putBlocks (this=0x1246668,
     buf=<optimized out>, frame_cnt=1024)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDevice.cpp:282
#52 0xb6e6d74c in Async::AudioDeviceAlsa::audioReadHandler (this=0x1246668,
     watch=0x128f6e8, revents=<optimized out>)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:412
#53 0xb6e6dcda in operator() (this=<optimized out>, _A_a1=<optimized out>,
     _A_a2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1917
#54 operator()<Async::FdWatch* const&, const short unsigned int&> (
     this=<optimized out>, _A_arg1=<optimized out>, _A_arg2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:103
#55 sigc::internal::slot_call2<sigc::bound_mem_functor2<void, Async::AudioDeviceAlsa, Async::FdWatch*, unsigned short>, void, Async::FdWatch*, unsigned short>::call_it (rep=<optimized out>, a_1=<optimized out>, a_2=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/slot.h:173
#56 0xb6e6e4fc in emit (_A_a2=@0xbed3dece: 1, _A_a1=@0xbed3dea4: 0x128f6e8,
---Type <return> to continue, or q <return> to quit---
     impl=0x128d228) at /usr/include/sigc++-2.0/sigc++/signal.h:1252
#57 emit (_A_a2=@0xbed3dece: 1, _A_a1=@0xbed3dea4: 0x128f6e8,
     this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2891
#58 operator() (_A_a2=@0xbed3dece: 1, _A_a1=@0xbed3dea4: 0x128f6e8,
     this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2899
#59 Async::AudioDeviceAlsa::AlsaWatch::readEvent (this=<optimized out>,
     watch=0x128f6e8)
     at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:154
#60 0xb6e6dcb4 in operator() (this=<optimized out>, _A_a1=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1851
#61 operator()<Async::FdWatch* const&> (this=<optimized out>,
     _A_arg1=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:84
#62 sigc::internal::slot_call1<sigc::bound_mem_functor1<void, Async::AudioDeviceAlsa::AlsaWatch, Async::FdWatch*>, void, Async::FdWatch*>::call_it (
     rep=<optimized out>, a_1=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/functors/slot.h:137
#63 0xb6e84038 in emit (_A_a1=@0x127b6a4: 0x128f6e8, impl=0x1256998)
     at /usr/include/sigc++-2.0/sigc++/signal.h:1010
#64 emit (_A_a1=@0x127b6a4: 0x128f6e8, this=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/signal.h:2781
#65 operator() (_A_a1=@0x127b6a4: 0x128f6e8, this=<optimized out>)
     at /usr/include/sigc++-2.0/sigc++/signal.h:2789
#66 Async::CppApplication::exec (this=0xbed3e140)
     at /home/cubie/svxlink/src/async/cpp/AsyncCppApplication.cpp:277
#67 0x00039f9c in main (argc=<optimized out>, argv=<optimized out>)
     at /home/cubie/svxlink/src/svxlink/remotetrx/remotetrx.cpp:558
(gdb)

Is it not possible to avoid the recursive call loops that apparently occur during error handling?

Rob

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Tobias Blomberg
On Monday 11 May 2015 18:18:21 Rob Janssen wrote:

> SM0SVX wrote:
> > On Sunday 10 May 2015 23:34:04 Rob Janssen wrote:
> >> Rob Janssen wrote:
> >>> Ok I am recompiling it now and will run it with core dump enabled on a
> >>> system where it sometimes fails.
> >>>
> >>> Rob
> >>
> >> I have tested it on a new receiver we recently deployed and which has a
> >> terrible internet connection. (still to be checked)
> >>
> >> Someone was being received on this receiver and probably there was
> >> another
> >> pause in the link. Result:
> >>
> >> RxNOS: SetMuteState(CONTENT)
> >> RxNOS: SetMuteState(NONE)
> >> RxNOS: SetMuteState(NONE)
> >> NetUplinkRX_NOS: Client disconnected: 44.137.42.60:56414
> >> RxNOS: The squelch is CLOSED (51.0691)
> >> *** ERROR: TCP transmit error in NetUplink "NetUplinkRX_NOS": Resource
> >> temporarily unavailable. Segmentation fault (core dumped)
> >>
> >> Core was generated by `/usr/bin/remotetrx
> >> --config=/etc/svxlink/remotetrx.conf --pidfile=/var/run/remo'. Program
> >> terminated with signal 11, Segmentation fault.
> >> #0  Async::TcpConnection::disconnect (this=0x0)
> >>
> >>       at /home/cubie/svxlink/src/async/core/AsyncTcpConnection.cpp:215
> >>
> >> 215       delete wr_watch;
> >
> > Ooops, sorry. That was due to my last bugfix. Instead of fixing it I added
> > a new bug. New code in master now.
> >
> > The stack trace look a little bit strange, probably due to optimizations.
> > Is it possible for you to run the software without optimizations turned
> > on or will it be too slow then?
> >
> > The second issue I need to look at further. It probably only (mostly?)
> > happen when quitting the application so not as serious as the first one.
> >
> > 73's de SM0SVX / Tobias
>
> I had started the new master code interactive now (but still a
> RelWithDebugInfo compile) and it crashes in the same way as I mentioned
> after the ^C but this time a while after the connection had failed and was
> re-established:

OK, so it's not just a problem when pressing Ctrl-C but can happen
spontaneously?


> RxNOS: The squelch is OPEN (28.3958)
> RxNOS: SetMuteState(NONE)
> RxNOS: The squelch is CLOSED (21.5164)
> RxNOS: The squelch is OPEN (10.3608)
> *** ERROR: TCP transmit buffer overflow in NetUplink NetUplinkRX_NOS.
> NetUplinkRX_NOS: Client disconnected: 44.137.42.60:55259
> RxNOS: The squelch is CLOSED (21.5961)
> RxNOS: The squelch is OPEN (21.7056)
> RxNOS: The squelch is CLOSED (26.2238)
> NetUplinkRX_UTR: Client disconnected: 44.137.42.10:35580
> NetUplinkRX_UTR: Client connected: 44.137.42.10:36362
> RxUTR: SetMuteState(NONE)
> NetUplinkRX_UTR: Using CODEC "OPUS" to encode RX audio
> ------ Opus encoder parameters ------
> Frame size           = 320
> Complexity           = 10
> Bitrate              = 32000
> VBR                  = YES
> Constrained VBR      = YES
> Maximum audio bw     = MEDIUMBAND
> Audio bw             = FULLBAND
> Signal type          = VOICE
> Application type     = AUDIO
> Inband FEC           = NO
> Expected Packet Loss = 0%
> DTX                  = NO
> --------------------------------------
> NetUplinkRX_NOS: Client connected: 44.137.42.60:57534
> RxNOS: SetMuteState(NONE)
> NetUplinkRX_NOS: Using CODEC "OPUS" to encode RX audio
> ------ Opus encoder parameters ------
> Frame size           = 320
> Complexity           = 10
> Bitrate              = 32000
> VBR                  = YES
> Constrained VBR      = YES
> Maximum audio bw     = MEDIUMBAND
> Audio bw             = FULLBAND
> Signal type          = VOICE
> Application type     = AUDIO
> Inband FEC           = NO
> Expected Packet Loss = 0%
> DTX                  = NO
> --------------------------------------
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: SetMuteState(CONTENT)
> RxNOS: SetMuteState(NONE)
> RxNOS: The squelch is OPEN (19.9524)
> remotetrx: /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168: int
> Async::AudioSource::sinkWriteSamples(const float*, int): Assertion `len >
> 0' failed. Aborted (core dumped)
>
> This may be the same scenario as I have posted before, usually we run it as
> a daemon and with logfile, so no Assert failed/Aborted messages and no core
> dumps. It would be nice when the error logging construct is somehow
> modified to pass assertion messages to the logfile...

That would be nice, wouldn't it :-)


> The stack backtrace is again very deep:
>
> Program terminated with signal 6, Aborted.
> #0  0xb6afaf96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> (gdb) bt
> #0  0xb6afaf96 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> #1  0xb6b08f8a in raise () from /lib/arm-linux-gnueabihf/libc.so.6
> #2  0xb6b0b428 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
> #3  0xb6b0401e in __assert_fail () from /lib/arm-linux-gnueabihf/libc.so.6
> #4  0xb6e5c710 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:168
> #5  0xb6e5cc0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #6  Async::AudioProcessor::writeFromBuf (this=0x1255228)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #7  0xb6e5cd56 in Async::AudioProcessor::writeSamples (this=0x1255228,
>      samples=0xbed3bab8, len=65)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:144
> #8  0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #9  0xb6e642d2 in Async::AudioDelayLine::writeSamples (this=0x1253c80,
>      samples=0x1251ba4, count=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDelayLine.cpp:240
> #10 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #11 0x000628b4 in writeSamples (count=65, samples=0x1251ba4, this=0x1253a58)
> at /home/cubie/svxlink/src/build/include/AsyncAudioPassthrough.h:143 #12
> Async::AudioStreamStateDetector::writeSamples (this=0x1253a58,
> samples=0x1251ba4, count=65)
>      at
> /home/cubie/svxlink/src/build/include/AsyncAudioStreamStateDetector.h:142
> #13 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #14 0x00041958 in Async::AudioValve::writeSamples (this=0x1253a90,
>      samples=<optimized out>, count=<optimized out>)
>      at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
> #15 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #16 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x1251ba4,
> this=0x1251b58) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #17
> writeSamples (len=65, samples=0x1251ba4, this=0x1252090)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #18 Async::AudioSplitter::writeSamples (this=0x1252090, samples=0x1251ba4,
>      len=65) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #19
> 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #20 0xb6e5cc0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #21 Async::AudioProcessor::writeFromBuf (this=0x1251b80)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #22 0xb6e5ce04 in Async::AudioProcessor::writeSamples (this=0x1251b80,
>      samples=0x12514c8, len=0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #23 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #24 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x12513c4,
> this=0x1251b30) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #25
> writeSamples (len=65, samples=0x12513c4, this=0x1251ae0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #26 Async::AudioSplitter::writeSamples (this=0x1251ae0, samples=0x12513c4,
>      len=65) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #27
> 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #28 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x12513c4,
> this=0x1251850) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #29
> writeSamples (len=65, samples=0x12513c4, this=0x1251800)
> ---Type <return> to continue, or q <return> to quit---
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #30 Async::AudioSplitter::writeSamples (this=0x1251800, samples=0x12513c4,
>      len=65) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #31
> 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #32 0xb6e5cc0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #33 Async::AudioProcessor::writeFromBuf (this=0x12513a0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #34 0xb6e5ce04 in Async::AudioProcessor::writeSamples (this=0x12513a0,
>      samples=0x1246838, len=0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #35 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #36 0x00041958 in Async::AudioValve::writeSamples (this=0x1250b08,
>      samples=<optimized out>, count=<optimized out>)
>      at /home/cubie/svxlink/src/build/include/AsyncAudioValve.h:240
> #37 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #38 0xb6e63f84 in sinkWriteSamples (len=65, samples=0x1246734,
> this=0x1251368) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:142 #39
> writeSamples (len=65, samples=0x1246734, this=0x1250a38)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:350
> #40 Async::AudioSplitter::writeSamples (this=0x1250a38, samples=0x1246734,
>      len=65) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioSplitter.cpp:329 #41
> 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized out>,
> samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #42 0xb6e5cc0a in writeFromBuf (this=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:322
> #43 Async::AudioProcessor::writeFromBuf (this=0x1246710)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:312
> #44 0xb6e5ce04 in Async::AudioProcessor::writeSamples (this=0x1246710,
>      samples=0x1250948, len=2)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioProcessor.cpp:185
> #45 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #46 0xb6e65c4c in writeSamplesFromFifo (this=0x124f8e0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:380
> #47 Async::AudioFifo::writeSamplesFromFifo (this=0x124f8e0)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:365
> #48 0xb6e65d8c in writeSamples (count=1024, samples=0xbed3be28,
> this=0x124f8e0) at
> /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:279 #49
> Async::AudioFifo::writeSamples (this=0x124f8e0, samples=0xbed3be28,
> count=1024) at /home/cubie/svxlink/src/async/audio/AsyncAudioFifo.cpp:225
> #50 0xb6e5c6f6 in Async::AudioSource::sinkWriteSamples (this=<optimized
> out>, samples=<optimized out>, len=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioSource.cpp:174
> #51 0xb6e62718 in Async::AudioDevice::putBlocks (this=0x1246668,
>      buf=<optimized out>, frame_cnt=1024)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDevice.cpp:282
> #52 0xb6e6d74c in Async::AudioDeviceAlsa::audioReadHandler (this=0x1246668,
>      watch=0x128f6e8, revents=<optimized out>)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:412
> #53 0xb6e6dcda in operator() (this=<optimized out>, _A_a1=<optimized out>,
>      _A_a2=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1917
> #54 operator()<Async::FdWatch* const&, const short unsigned int&> (
>      this=<optimized out>, _A_arg1=<optimized out>, _A_arg2=<optimized out>)
> at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:103 #55
> sigc::internal::slot_call2<sigc::bound_mem_functor2<void,
> Async::AudioDeviceAlsa, Async::FdWatch*, unsigned short>, void,
> Async::FdWatch*, unsigned short>::call_it (rep=<optimized out>,
> a_1=<optimized out>, a_2=<optimized out>) at
> /usr/include/sigc++-2.0/sigc++/functors/slot.h:173
> #56 0xb6e6e4fc in emit (_A_a2=@0xbed3dece: 1, _A_a1=@0xbed3dea4: 0x128f6e8,
> ---Type <return> to continue, or q <return> to quit---
>      impl=0x128d228) at /usr/include/sigc++-2.0/sigc++/signal.h:1252
> #57 emit (_A_a2=@0xbed3dece: 1, _A_a1=@0xbed3dea4: 0x128f6e8,
>      this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2891
> #58 operator() (_A_a2=@0xbed3dece: 1, _A_a1=@0xbed3dea4: 0x128f6e8,
>      this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/signal.h:2899
> #59 Async::AudioDeviceAlsa::AlsaWatch::readEvent (this=<optimized out>,
>      watch=0x128f6e8)
>      at /home/cubie/svxlink/src/async/audio/AsyncAudioDeviceAlsa.cpp:154
> #60 0xb6e6dcb4 in operator() (this=<optimized out>, _A_a1=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/mem_fun.h:1851
> #61 operator()<Async::FdWatch* const&> (this=<optimized out>,
>      _A_arg1=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/adaptors/adaptor_trait.h:84
> #62 sigc::internal::slot_call1<sigc::bound_mem_functor1<void,
> Async::AudioDeviceAlsa::AlsaWatch, Async::FdWatch*>, void,
> Async::FdWatch*>::call_it ( rep=<optimized out>, a_1=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/functors/slot.h:137
> #63 0xb6e84038 in emit (_A_a1=@0x127b6a4: 0x128f6e8, impl=0x1256998)
>      at /usr/include/sigc++-2.0/sigc++/signal.h:1010
> #64 emit (_A_a1=@0x127b6a4: 0x128f6e8, this=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/signal.h:2781
> #65 operator() (_A_a1=@0x127b6a4: 0x128f6e8, this=<optimized out>)
>      at /usr/include/sigc++-2.0/sigc++/signal.h:2789
> #66 Async::CppApplication::exec (this=0xbed3e140)
>      at /home/cubie/svxlink/src/async/cpp/AsyncCppApplication.cpp:277
> #67 0x00039f9c in main (argc=<optimized out>, argv=<optimized out>)
>      at /home/cubie/svxlink/src/svxlink/remotetrx/remotetrx.cpp:558
> (gdb)
>
> Is it not possible to avoid the recursive call loops that apparently occur
> during error handling?

That the stack trace is deep does not have to indicate that it is a problem.
It could be a problem but I don't think it's pure recursion you are seeing.
There are a lot of audio pipe objects chained together and all of them have a
hierarchy of inheritance. Often you see AudioSource in the trace at many
places but that is because it's the base class of all audio source objects.
It's not because it's calling itself recursively. It's different objects.

73's de SM0SVX / Tobias


>
> Rob
>
> ----------------------------------------------------------------------------
> -- One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Svxlink-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/svxlink-devel


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
SM0SVX wrote:
> I had started the new master code interactive now (but still a
> RelWithDebugInfo compile) and it crashes in the same way as I mentioned
> after the ^C but this time a while after the connection had failed and was
> re-established:
> OK, so it's not just a problem when pressing Ctrl-C but can happen
> spontaneously?
yes, in this case that is what happened.  but it is not always the same.

>> It would be nice when the error logging construct is somehow
>> modified to pass assertion messages to the logfile...
> That would be nice, wouldn't it :-)

I think it would be possible to make a custom assert macro, or to change the way the
logging works.   When the logfile is just opened on stderr it will work OK.  However that
will mean each log output will have to insert the timestamp itself, not via the mechanism
currently used on the loopback pipe.  The advantage would be that when log is to the
screen, it is timestamped as well.

>
>
> That the stack trace is deep does not have to indicate that it is a problem.
>

Ok...  I thought maybe the reason it crashes is that at some point it is calling back to the
same function that was already running in an error.
Maybe some of the asserts should be changed into errors that output a message and
return a returncode, instead of aborting the program.

Rob

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Tobias Blomberg
On Sunday 17 May 2015 20:19:57 Rob Janssen wrote:

> SM0SVX wrote:
> > I had started the new master code interactive now (but still a
> > RelWithDebugInfo compile) and it crashes in the same way as I mentioned
> > after the ^C but this time a while after the connection had failed and was
> > re-established:
> > OK, so it's not just a problem when pressing Ctrl-C but can happen
> > spontaneously?
>
> yes, in this case that is what happened.  but it is not always the same.
>
> >> It would be nice when the error logging construct is somehow
> >> modified to pass assertion messages to the logfile...
> >
> > That would be nice, wouldn't it :-)
>
> I think it would be possible to make a custom assert macro, or to change the
> way the logging works.   When the logfile is just opened on stderr it will
> work OK.  However that will mean each log output will have to insert the
> timestamp itself, not via the mechanism currently used on the loopback
> pipe.  The advantage would be that when log is to the screen, it is
> timestamped as well.

Yes, but it's a lot of work to change it...


> > That the stack trace is deep does not have to indicate that it is a
> > problem.
> Ok...  I thought maybe the reason it crashes is that at some point it is
> calling back to the same function that was already running in an error.
> Maybe some of the asserts should be changed into errors that output a
> message and return a returncode, instead of aborting the program.

I use assertions in places where errors should normally not happen. Continued
execution could cause problems or recovering nicely could be a lot of work.
Using return codes to handle errors in the audio pipe is hard. All components
would have to be rewritten to handle it.

Actually, I have a feeling that this bug is caused by a function returning -1
that should normally only return >= 0.

73's de SM0SVX / Tobias

>
> Rob


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
Reply | Threaded
Open this post in threaded view
|

Re: debugging a new problem

Rob Janssen
SM0SVX wrote:
> On Sunday 17 May 2015 20:19:57 Rob Janssen wrote:
>> I think it would be possible to make a custom assert macro, or to change the
>> way the logging works.   When the logfile is just opened on stderr it will
>> work OK.  However that will mean each log output will have to insert the
>> timestamp itself, not via the mechanism currently used on the loopback
>> pipe.  The advantage would be that when log is to the screen, it is
>> timestamped as well.
> Yes, but it's a lot of work to change it...

Well, that should not be too difficult.  I made changes like that to projects before, one
can write a logerr() function that takes a printf format and use a simple shell script like:

for f in `grep -lR 'fprintf(stderr,' .`
do
    sed -i -e 's/fprintf(stderr,/logerr(/' $f
done

(and similar for normal printf when using '\<printf(' matching)

I don't have experience doing that with C++, no idea if it is possible to write a replacement
for cout and cerr that will use the same syntax but behaves differently.

Rob

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Svxlink-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/svxlink-devel
12