Month: May 2014

My new server

Just bought a 2nd hand HP server (8core + 16GB ram), cost me $2630 HKD. One 1TB 2.5″ hard disk cost me $450. Total $3080. It reboot only take 20 seconds, dell needs 5 mins (CRAZY!!!)

 

/root>cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping	: 10
microcode	: 0xa07
cpu MHz		: 2500.059
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority
bogomips	: 5000.11
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

2nd hand server

read count : 615

Analyst android ELF dependency-graph by elf-dependency-walker

Someone asked me in email about using elf-dependency-walker to analyst library or executable in linux. I found out android is using ELF format for its libraries or executables too. So the rest is simple. Please download the android emulator, within the directory there is the system image, which contain the file system.

system image
system image

You can see in the above graph, it is a file called “system.img”. It is the file system image in ext4 format. So you can mount it by this command:

mkdir temp
mount -o loop system.img temp
mount it
mount it

Now, cd to the temp directory, you can see all android files in there.

change the lookup directory path in elf-dependency-wwalker
change the lookup directory path in elf-dependency-wwalker

Just start elf-dependency-walker by “java -jar elf-dependency-walker.jar”. You can download it in https://code.google.com/p/elf-dependency-walker. Click on “Setting”, and place the full path of your mounted directory (The directory you mounted to your android fs) in it.

select any executable/library from the android fs
select any executable/library from the android fs

Click “Analyst” button, choose any file (must be elf format file).

eld chart is generated
eld chart is generated

It will analyst the dependency for you.

read count : 719

solved : compile error in ffmpeg “error: invalid combination of opcode and operands”‏

I am using ubuntu 12.04 64 bits and trying compile ffmpeg and got this error:

libavfilter/x86/vf_pullup.asm:59: error: invalid combination of opcode and operands
libavfilter/x86/vf_pullup.asm:138: error: invalid combination of opcode and operands
libavfilter/x86/vf_pullup.asm:175: error: invalid combination of opcode and operands

To solve this, add “–disable-mmx” to your ./configure

read count : 979

sox and ffmpeg (convert hls into live hls) usage

These are the basic usage of SOX

# Create background noise profile from mp3
/usr/bin/sox noise.mp3 -n noiseprof noise.prof
 
# Remove noise from mp3 using profile
/usr/bin/sox input.mp3 output.mp3 noisered noise.prof 0.21
 
# Remove silence from mp3
/usr/bin/sox input.mp3 output.mp3 silence -l 1 0.3 5% -1 2.0 5%
 
# Remove noise and silence in a single command
/usr/bin/sox input.mp3 output.mp3 noisered noise.prof 0.21 silence -l 1 0.3 5% -1 2.0 5%
 
# Batch process files
/usr/bin/find . -type f -name "*.mp3" -mmin +30 -exec sox -S --multi-threaded -buffer 131072 {} /path/to/output/{} noisered noise.prof 0.21 silence -l 1 0.3 5% -1 2.0 5% \;
 
# Remove insignificant files
/usr/bin/find . -type f -name "*.mp3" -mmin +30 -size -500k -delete

This script will convert ffmpeg HLS into “live streaming HLS”

#!/bin/bash

x=0
while true; do
	#cat out.m3u8|sed 's/YES/NO/' | sed 's/#EXT-X-ENDLIST//'> temp
	line=`wc -l out.m3u8|awk '{print $1}'`
	line=$(($line-5))
	line=$(($line/2))
	echo $line

	head -5 out.m3u8 |sed 's/YES/NO/' | sed 's/#EXT-X-ENDLIST//' | sed "s/#EXT-X-MEDIA-SEQUENCE:0/#EXT-X-MEDIA-SEQUENCE:$line/" > temp
	#echo "#EXTM3U">temp
	#echo "#EXT-X-VERSION:3">>temp;
	#echo "#EXT-X-TARGETDURATION:2">>temp;
	#echo "#EXT-X-MEDIA-SEQUENCE:$x">>temp;
	tail -16 out.m3u8 >> temp;
	#echo "#EXTINF:1,">>temp;
	#echo "dummy.mp3">>temp;

	tail -8 out.m3u8 | grep mp3 | while read a; do
		sox $a c$a noisered noise.prof 0.21
	done	

	sed -i 's/a0/ca0/' temp

	mv temp out2.m3u8
	x=$(($x+1))
	#sleep 2
done

The above script has to run with ffmpeg:
/root/download/ffmpeg-2.1.4/ffmpeg -f alsa -ac 2 -i hw:0,0 -strict experimental -acodec libmp3lame -map 0 -f segment -segment_list out.m3u8 -segment_format libmp3lame -segment_time 0.5 -segment_list_flags +live ‘a%05d.mp3’

read count : 663

newlib’s reentrant system calls will map to non-reentrant by default

newlib’s reentrant system calls will map to non-reentrant by default. The reentrant version of open() can be found in newlib-2.1.0/newlib/libc/reent/openr.c

int
_DEFUN (_open_r, (ptr, file, flags, mode),
     struct _reent *ptr _AND
     _CONST char *file _AND
     int flags _AND
     int mode)
{
  int ret;

  errno = 0;
  if ((ret = _open (file, flags, mode)) == -1 && errno != 0)
    ptr->_errno = errno;
  return ret;
}

As you can see, it will call _open which is defined in newlib-2.1.0/newlib/libgloss/libnosys/open.c . So if you don’t write your reentrant functions, it will jump to non-reentrant by default.

For OS developer want to implement reentrant and non-reentrant function, focus on these two directories newlib-2.1.0/newlib/libc/reent/ and newlib-2.1.0/newlib/libgloss/libnosys

Summary:

Let use open() as an example, newlib defined 4 open() function, they are:

1. newlib-2.1.0/libgloss/libnosys/open.c <– defines open()

2. newlib-2.1.0/newlib/libc/reent/openr.c <– defines _open_r() , which will call to _open()

3. newlib-2.1.0/newlib/libc/syscalls/sysopen.c <– defined open() , which will call to _open_r()

4. newlib-2.1.0/libgloss/open.c <– define open(), and don’t call anything

 

!!! If you just add “syscall_dir=syscalls” to configure.host, only 1,2,4 will be compiled.

What is -DMISSING_SYSCALL_NAMES

“-DMISSING_SYSCALL_NAMES” is just use to rename _open to open, _lseek to sleek and etc… Take a look the file newlib-2.1.0/libgloss/libnosys/open.c:

#include "config.h"
#include <_ansi.h>
#include <_syslist.h>
#include <errno.h>
#undef errno
extern int errno;
#include "warning.h"

int
_DEFUN (_open, (file, flags, mode),
        char *file  _AND
        int   flags _AND
        int   mode)
{
  errno = ENOSYS;
  return -1;
}

stub_warning(_open)

It try to define _open, and _open is defined in _syslist.h:

#ifdef MISSING_SYSCALL_NAMES
#define _close close
#define _execve execve
#define _fcntl fcntl
#define _fork fork
#define _fstat fstat
#define _getpid getpid
#define _gettimeofday gettimeofday
#define _isatty isatty
#define _kill kill
#define _link link
#define _lseek lseek
#define _mkdir mkdir
#define _open open
#define _read read
#define _sbrk sbrk
#define _stat stat
#define _times times
#define _unlink unlink
#define _wait wait
#define _write write
#endif

So if you add “MISSING_SYSCALL_NAMES” to compile option, _open will be rename to open. If you added MISSING_SYSCALL_NAMES, try to disassemble the libnosys.a by command “objdump -dS ./i586-peter-elf/libgloss/libnosys/libnosys.a”, you can see _open is compiled as open:

00000000 <open>:
_DEFUN (_open, (file, flags, mode),

What is -DREENTRANT_SYSCALLS_PROVIDED

If you add it to your compile option in configure.host, all file in newlib-2.1.0/newlib/libc/reent/ will not be compile

Ways to add stubs

Based on this document (Howto: Porting newlib – A Simple Guide – Embecosm), there are total 4 alternatives for specifying how to implement the stubs

1. only “syscall_dir=syscalls”

Define namespace clean versions of the system calls by prefixing them with ‘_’ (eg: _open, _close, etc.). Technically, there won’t be true reentrancy at the syscall level, but the library will be namespace clean. When you do this, set “syscall_dir” to “syscalls” in configure.host.

or32-*-*)
syscall_dir=syscalls
;;

_open_r will be compiled into libc.a and _open will be in libnosys.a

2. “syscall_dir=syscalls” and “-DREENTRANT_SYSCALLS_PROVIDED”

Define the reentrant versions of the syscalls directly. (eg: _open_r, _close_r, etc.). Please keep the namespace clean. When you do this, set “syscall_dir” to “syscalls” and add -DREENTRANT_SYSCALLS_PROVIDED to newlib_cflags in configure.host.

If you add these two options in configure.host

or32-*-*)
syscall_dir=syscalls
newlib_cflags="${newlib_cflags} -DREENTRANT_SYSCALLS_PROVIDED"
;;

_open_r will *NOT* be compiled in libc.a and _open will be in libnosys.a. If you look at file newlib-2.1.0/newlib/libc/reent/openr.c, there is “#ifndef REENTRANT_SYSCALLS_PROVIDED”, so if you provide “REENTRANT_SYSCALLS_PROVIDED”, nothing will be compile.

3. “-DMISSING_SYSCALL_NAMES”

Define or otherwise provide the regular versions of the syscalls (eg: open, close, etc.). The library won’t be reentrant nor namespace clean, but at least it will work. When you do this, add -DMISSING_SYSCALL_NAMES to newlib_cflags in configure.host.

or32-*-*)
      newlib_cflags="${newlib_cflags} -DMISSING_SYSCALL_NAMES"
      ;;

_open_r will be compiled in libc.a and open will be in libnosys.a

4. “-DMISSING_SYSCALL_NAMES” and -DREENTRANT_SYSCALLS_PROVIDED”

Define or otherwise provide the regular versions of the sys calls, and do not supply functional interfaces for any of the reentrant calls. With this method, the reentrant syscalls are redefined to directly call the regular system call without the reentrancy argument. When you do this, specify both -DREENTRANT_SYSCALLS_PROVIDED and -DMISSING_SYSCALL_NAMES via newlib_cflags in configure.host and do not specify “syscall_dir”.

or32-*-*)
      newlib_cflags="${newlib_cflags} -DMISSING_SYSCALL_NAMES"
      newlib_cflags="${newlib_cflags} -DREENTRANT_SYSCALLS_PROVIDED"
      ;;

_open_r will *NOT* be compiled in libc.a and open will be in libnosys.a

Something need to be care

There are two case-switches (case “${host}”) in configure.host, add your machine/target to the second one, because the second-case-switch got a “*)”, otherwise it will always flow to “*)”

read count : 1644

gcc, binutils and newlib can all be compiled at once.

gcc, binutils and newlib can all be compiled at once. This article http://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.pdf said gcc, binutils and newlib are from the same source tree. Using the following script can linkup all files into a single directory and compile all of them at once.

#!/bin/bash
component_dirs='binutils-2.24 gcc-4.9.0 newlib-2.1.0'
unified_src=srcw
mkdir ${unified_src}
cd ${unified_src}
ignore_list=". .. CVS .svn"

for srcdir in ${component_dirs}
do
 echo "Component: $srcdir"
 case srcdir
 in
 /* | [A-Za-z]:[\\/]*)
 ;;

 *)
 srcdir="../${srcdir}"
 ;;
 esac

 files=`ls -a ${srcdir}`

 for f in ${files}
 do
 found=

 for i in ${ignore_list}
 do
 if [ "$f" = "$i" ]
 then
 found=yes
 fi
 done

 if [ -z "${found}" ]
 then
 echo "$f ..linked"
 ln -s ${srcdir}/$f .
 fi
 done
 ignore_list="${ignore_list} ${files}"
done
cd ..

Just put the directories (binutils-2.24 , gcc-4.9.0 , newlib-2.1.0) and the above script in the same directory. Run the above script, it will create a directory “srcw”, it will contains anything. Then create “build” directory, go into it and execute “../srcw/configure –target=i586-peter-elf –enable-languages=c –prefix=/somewhere –with-newlib” .

After all you can compile install it by :

make all-build all-binutils all-gas all-ld all-gcc all-target-newlib all-target-libgloss
make install-binutils install-gas install-ld install-gcc install-target-newlib install-target-libgloss

All newlib, binutils and gcc will be build at once

One thing that pdf is wrong is : latest gcc doesn’t support “make all-build”

read count : 489