2007年10月28日 星期日

十年

人總是在失去後才覺得珍貴。

這幾天正好是我們認識十週年,雖然分手好一段時間了,而我們又各自有了新的戀情。
聽到這首歌,份外有感覺。

對不起,我們錯過了…


十年 - 陳奕迅 曲︰陳小霞 詞︰林夕 編︰陳輝陽
如果那兩個字沒有顫抖
我不會發現我難受
怎麼說出口 也不過是分手
如果對於明天沒有要求
牽牽手就像旅遊 成千上萬個門口
總有一個人要先走

懷抱既然不能逗留 何不在離開的時候
一邊享受 一邊淚流

十年之前 我不認識你
你不屬於我 我們還是一樣
陪在一個陌生人左右
走過漸漸熟悉的街頭

十年之後 我們是朋友
還可以問候 只是那種溫柔
再也找不到擁抱的理由
情人最後難免淪為朋友

直到和你做了多年朋友
才明白我的眼淚 不是為你而流
也為別人而流

2007年10月22日 星期一

Stack Dump Debug Tip

In developing embedded system, developer may encounter an ambiguous condition that the system so limited that cannot run gdb, and the network is also too unstable to run gdb-server. And the rootfs is too small that core dump is not available. In such miserable condition there is still a powerful debug skill: stack dump.

Before using this skill, we have to know how a program runs. The stack's bottom is at higher address in memory, and it growth toward lower address.


And program accesses memory from low to high. That is we can use printf("%x %x"); to print out the data in stack.

Another important tip is CFLAGS += -finstrument-functions. This flag make us can insert code when entering and exiting a function. This way, we can construct our own stack dump without the help of gdb. If you turn the flag -finstrument-functions on, you should implements the following two functions:
void __cyg_profile_func_enter(void *func,void *caller) __attribute__((__no_instrument_function__));
void __cyg_profile_func_exit(void *func,void *caller) __attribute__((__no_instrument_function__));

These functions are designed for profiling and analysing code performance and coverage. In this case use use it to create a function stack.

We can use signal() to set the signal handler to dump the stack when a signal received.

Here are the demo codes:

----------------------------------------------------------------
### File: Makefile
CFLAGS=-finstrument-functions -g2
CC=gcc

all:
gcc $(CFLAGS) -o stackdemo stackdemo.c stack_util.c
----------------------------------------------------------------
/** @file stack_util.h
* @auther Tick
* @License GPL
*/


#ifndef __TICK_DEMO_STACK___
#define __TICK_DEMO_STACK___

void setSignalHandler();
void SignalHandler(int) __attribute__((__no_instrument_function__));
void DumpStack(int sig) __attribute__((__no_instrument_function__));
void __cyg_profile_func_enter(void *func,void *caller) __attribute__((__no_instrument_function__));
void __cyg_profile_func_exit(void *func,void *caller) __attribute__((__no_instrument_function__));

#endif
----------------------------------------------------------------
/** @file stack_util.c
* @auther Tick
* @License GPL
*/


#include "stack_util.h"

#include <signal.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>

#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/ioctl.h>


#define _STACK_BUFFER_SIZE 4096
void *__stack_buffer_[_STACK_BUFFER_SIZE];
int __stack_pid_buffer[_STACK_BUFFER_SIZE];
int __stack_buffer_index__private__ =0;
volatile int __stack_spin_lock__=0;
inline void dumpstack() {
int i,max=__stack_buffer_index__private__;
printf("Stack Trace:\n");
for (i=0;i<__stack_buffer_index__private__;i++) {
printf("[%3d] pid:[%3d] 0x%08x\n",i,__stack_pid_buffer[i],__stack_buffer_[i]);
}
}

void setSignalHandler() {
int ans;
struct sigaction act,oldact;
act.sa_handler=DumpStack;
act.sa_flags= SA_ONESHOT | SA_NOMASK;
//act.sa_flags= SA_RESETHAND;
ans=sigaction(SIGUSR1,&act,&oldact);
if (ans) {
printf("Set SIGUSR1 Failed!!!!!\n");
printf ("ErrorNumber: %d\n",errno);
}else {
printf("Set SIGUSR1 OK!!!!!\n");
}

signal(SIGHUP,SignalHandler);
signal(SIGINT,SignalHandler);
signal(SIGQUIT,SignalHandler);
signal(SIGILL,SignalHandler);
signal(SIGKILL,SignalHandler);
signal(SIGSEGV,SignalHandler);
signal(SIGTERM,SignalHandler);
signal(SIGPWR,SignalHandler);
}

void DumpStack(int sig) {
struct sigaction act,oldact;
act.sa_handler=DumpStack;
act.sa_flags= SA_ONESHOT | SA_NOMASK;
sigaction(SIGUSR1,&act,&oldact);

printf("Stack:\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n");

printf ("Signal Type: %d\n",sig);
printf ("ErrorNumber: %d\n",errno);
printf ("PID: %d\n",getpid());
dumpstack();
}

void SignalHandler(int sig) {
if (sig==SIGTERM || sig==SIGKILL) {

//close device here
}
if (sig==SIGTERM) {
printf ("Signal Type: %d\n",sig);
printf ("PID: %d goes die\n",getpid());
fflush(stdout);
fflush(stderr);
sleep(1);
exit(0);
}

printf("Stack:\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x 0x%08x\n");

printf ("Signal Type: %d\n",sig);
printf ("ErrorNumber: %d\n",errno);
printf ("PID: %d\n",getpid());
printf ("\n");
dumpstack();
fflush(stdout);
fflush(stderr);
if (sig==SIGSEGV || sig==SIGILL) {
//exit(EXIT_FAILURE);
// If you want to kill whole tick when page fault happens
// please USE "exit(EXIT_FAILURE);"
_exit(129);
}
}


void __cyg_profile_func_enter(void *func,void *caller) {
while (__stack_spin_lock__) ;
__stack_spin_lock__=1;
if (__stack_buffer_index__private__ < _STACK_BUFFER_SIZE) {
__stack_buffer_[__stack_buffer_index__private__]=func;
__stack_pid_buffer[__stack_buffer_index__private__]=getpid();
__stack_buffer_index__private__++;
}
__stack_spin_lock__=0;
}
void __cyg_profile_func_exit(void *func,void *caller) {
while (__stack_spin_lock__) ;
__stack_spin_lock__=1;
int i,pid=getpid();
if (__stack_buffer_index__private__ > 0) {
for (i=__stack_buffer_index__private__-1;i >0;i--) {
if (__stack_pid_buffer[i]==pid) {
break;
}
}
for (;i<__stack_buffer_index__private__-1;i++) {
__stack_buffer_[i]=__stack_buffer_[i+1];
__stack_pid_buffer[i]=__stack_pid_buffer[i+1];
}
__stack_buffer_index__private__--;
}
__stack_spin_lock__=0;
}
----------------------------------------------------------------
/**
* @file stackdemo.c
* @author Tick
* @license GPL
*/

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <ctype.h>
#include "stack_util.h"

struct LALA {
int data;
int lala;
};

void run_me_will_crash_null_point () {
struct LALA *lala=NULL;
lala->lala++;
}

void run_me_will_crash_divid_zero() {
int a=1;
int b= 200/ --a;
}

void run_me_will_crash_buffer_overflow() {
char buf[4];
char *src;
src =(char *) malloc (64);
memset(src,'A',64);
memcpy(buf,src,64);
}


void prompt() {
printf("What way do you want to die?\n");
printf("1. segment fault\n");
printf("2. divid zero\n");
printf("3. buffer overflow\n");
printf("> ");
}

void do_die () {
char buf[128];
while(1) {
prompt();
fgets(buf,sizeof(buf),stdin);
if (isdigit(buf[0])) {
switch (buf[0]) {
case '1':
run_me_will_crash_null_point();
break;
case '2':
run_me_will_crash_divid_zero();
break;
case '3':
run_me_will_crash_buffer_overflow();
break;
default:
continue;
}
}
}
}

void showmaps() {
char buf[128];
sprintf(buf,"cat /proc/%d/maps",getpid());
system(buf);
}

void function_D() {do_die();}
void function_C() {function_D();}
void function_B() {function_C();}
void function_A() {function_B();}
int main (void) {
setSignalHandler();
showmaps();
function_A();
}
----------------------------------------------------------------


The following is the execution result:

----------------------------------------------------------------

tick@tock:~/work/demo>./stackdemo
Set SIGUSR1 OK!!!!!
08048000-0804a000 r-xp 00000000 08:03 127325 /home/tick/work/demo/stackdemo
0804a000-0804b000 rw-p 00001000 08:03 127325 /home/tick/work/demo/stackdemo
0804b000-08053000 rw-p 0804b000 00:00 0 [heap]
b7dc7000-b7dc8000 rw-p b7dc7000 00:00 0
b7dc8000-b7f0c000 r-xp 00000000 08:03 121895 /lib/tls/i686/cmov/libc-2.6.1.so
b7f0c000-b7f0d000 r--p 00143000 08:03 121895 /lib/tls/i686/cmov/libc-2.6.1.so
b7f0d000-b7f0f000 rw-p 00144000 08:03 121895 /lib/tls/i686/cmov/libc-2.6.1.so
b7f0f000-b7f12000 rw-p b7f0f000 00:00 0
b7f25000-b7f28000 rw-p b7f25000 00:00 0
b7f28000-b7f42000 r-xp 00000000 08:03 121912 /lib/ld-2.6.1.so
b7f42000-b7f44000 rw-p 00019000 08:03 121912 /lib/ld-2.6.1.so
bfc0f000-bfc24000 rw-p bfc0f000 00:00 0 [stack]
ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]
What way do you want to die?
1. segment fault
2. divid zero
3. buffer overflow
> 1
Stack:
0x00000001 0xbfc22948 0xffffe420 0x0000000b 0x00000033 0xc02f0000 0x0000007b 0x0000007b
0x00000000 0xb7f42ce0 0xbfc22948 0xbfc22920 0xb7f0dff4 0x00000000 0xb7f0dff4 0x00000000
0x0000000e 0x00000004 0x08048837 0x00000073 0x00210286 0xbfc22920 0x0000007b 0x00000000
0x00000000 0x00000004 0xb7f38068 0x00000011 0x00000008 0xb7f42ff4 0xb7f28468 0xb7f28000
0xbfc226e8 0xbfc22704 0x00000000 0xbfc22800 0xbfc226dc 0xbfc22768 0xb7df2b5a 0xbfc22818
0xbfc226e8 0x00000080 0x00000000 0x00000000 0xb6d55b68 0x00000000 0x00000000 0x00000000
0x00000000 0xb7f27db4 0xb7f303e2 0xb7f12000 0x00013425 0xb7f42ff4 0xbfc22aa4 0xb7f2b22b
0xb7f432a0 0xb7f43820 0x00000000 0x00000000 0x00000000 0x00000000 0xbfc2286c 0x00000000
0xbfc227d4 0xb7df2b13 0xbfc227e0 0xb7f0ef64 0x00000080 0x08048ec0 0x10000000 0xb7f27b38
0x00000004 0x080484f4 0x0d696910 0xb7f27b38 0xbfc2279c 0xb7f30c0b 0xb7dd9df6 0x080484ea
0xb7f43941 0x080484ea 0x0d696913 0xb7f27b48 0xbfc227bc 0xb7f30c0b 0xb7dd9e74 0x080484e0
0xb7f43941 0xb7f43934 0xb7dd0dac 0xbfc2000d 0xb7f35c99 0x080483e8 0xb7f43940 0xb7f42ff4
0xbfc22800 0xb7f27b0c 0xbfc22834 0xb7f30fc0 0xb7e2e0d4 0xb7e983b8 0xb7f0e440 0xb7f0dff4
0xb7f3efbc 0xbfc22800 0xbfc228e4 0x22494966 0x00000003 0xb7dcbc4c 0xb7dcada0 0xffffffff
0x22494966 0x0000000b 0xbfc22828 0x00000000 0x00000000 0x00000001 0x00000516 0xb7f27b48
0xb7f27858 0x08048467 0xb7dd485c 0x08048308 0x00000001 0xb7f42ff4 0xb7f43820 0xbfc228d8
0xbfc228f4 0xb7f31166 0x08048308 0xbfc228d8 0xb7f437c4 0x00000001 0xb7f27b48 0x00000001
0x00000000 0x00000001 0xffffffff 0xbfc22884 0xb7e2c179 0xb7f0e4e0 0xb7f25000 0x00000002
0xbfc228a4 0xbfc228e4 0xbfc22920 0xb7f43668 0x08048467 0x22494966 0x00000000 0x00000000
0x00000400 0xb7f0dff4 0xb7f0e440 0x00000000 0xbfc228b4 0xb7e2d50b 0xb7f0e440 0xb7f0e440
0xbfc228cc 0xb7e2e8d4 0xb7f0e440 0xb7e2b86f 0xb7f0dff4 0xb7f24001 0x00000001 0xb7e2206d
0xbfc22975 0xb7dd0dac 0xb7f27858 0x00000001 0xffffffff 0xb7f42ff4 0x08048308 0xb7f43668
0xbfc22930 0xb7f34e73 0xb7f43820 0xb7f27b48 0x00000001 0x00000001 0x00000000 0x0077b858
0x80cd0000 0xbfc22948 0x0804882d 0x08048814 0x08048ac2 0xb7f42ce0 0x00000000 0xbfc229f8
0xb7f3a660 0xb7f0f0c4 0xbfc22974 0xb7dec4d0 0x00000000 0xbfc229f8 0x08048ac2 0xbfc22974
0x00000080 0xb7f0e440 0xb7f27858 0x00000000 0x00000031 0xb7f42ff4 0x08048298 0xb7f43668
0xbf000a31 0xb7f34e73 0xb7f43820 0xb7f27b38 0x00000001 0x00000001 0x00000000 0x0804847c
0x00000006 0x00000000 0x0804acc4 0xb7dd106c 0xb7dff940 0xbfc229b4 0x08049039 0xbfc22a78
0x0000188d 0x00000000 0xb7f0dff4 0x00000001 0x0000188c 0xbfc22a78 0x08048b3f 0x08048ad8
0x08048c4c 0x0000188c 0x0d696910 0x00000000 0x080f0d30 0x08048e34 0x08048ff7 0xbfc22a98
0xff0a0000 0xbfc22a18 0x08048b70 0x08048b52 0x08048ba3 0xb7f42ff4 0x08048ff7 0xb7f43668
0xb7f0dff4 0xbfc22a38 0x08048ba3 0x08048b85 0x08048bd6 0x00000001 0x08048ff7 0x0804843b
0xb7f0dff4 0xbfc22a58 0x08048bd6 0x08048bb8 0x08048c09 0xb7f42ce0 0x08048ff7 0xbfc22a78
0xb7f0dff4 0xbfc22a78 0x08048c09 0x08048beb 0x08048c51 0xb7e5bc08 0xc0000000 0xb7ed2939
0xff0a0000 0xbfc22a98 0x08048c51 0x08048c1e 0xb7dde050 0xbfc22aa8 0x080490f9 0xb7f35800
0xbfc22ab0 0xbfc22b08 0xb7dde050 0xb7f42ce0 0x080490e0 0xbfc22b08 0xb7dde050 0x00000001
0xbfc22b34 0xbfc22b3c 0xb7f43820 0x00000000 0x00000001 0x00000001 0x00000000 0xb7f0dff4
0xb7f42ce0 0x00000000 0xbfc22b08 0x90556081 0xafc02a91 0x00000000 0x00000000 0x00000000
0xb7f3a660 0xb7dddf7d 0xb7f42ff4 0x00000001 0x08048790 0x00000000 0x080487b1 0x08048c1e
0x00000001 0xbfc22b34 0x080490e0 0x080490d0 0xb7f35800 0xbfc22b2c 0xb7f3fead 0x00000001
0xbfc23864 0x00000000 0xbfc23870 0xbfc23883 0xbfc238ba 0xbfc238ca 0xbfc238de 0xbfc238e9
0xbfc23938 0xbfc23953 0xbfc2398c 0xbfc2399e 0xbfc239a8 0xbfc23c23 0xbfc23c59 0xbfc23c89
Signal Type: 11
ErrorNumber: 22
PID: 6284

Stack Trace:
[ 0] pid:[6284] 0x08048c1e
[ 1] pid:[6284] 0x08048beb
[ 2] pid:[6284] 0x08048bb8
[ 3] pid:[6284] 0x08048b85
[ 4] pid:[6284] 0x08048b52
[ 5] pid:[6284] 0x08048a1e
[ 6] pid:[6284] 0x08048814
[ 7] pid:[6284] 0x08048c70

----------------------------------------------------------------
You can use "objdump -S -x stackdemo | less" to find out the address's corresponding function.

. If the function crashed in library, you can using the address - the address of library (shown in /proc/xxx/maps) and find out the offset. This offset is the crash point in library. (using objdump)