分享到plurk 分享到twitter 分享到facebook

版本 27577477994267ecaeb109e2bceed2ea69329f09

embedded/xenomai

Changes from 27577477994267ecaeb109e2bceed2ea69329f09 to c47d3ffc75895896a50c4ee7c31ae32eafbd5e02

---
title: Xenomai
categories: embedded, arm, raspberrypi, rpi, realtime, linux, rtos, xenomai, preempt_rt
...

協作者
=======
* 2015 年春季
  - 吳哲綱, 曾柏翔, 林展翔, 詹志鴻, 林建慶, 呂鴻,  楊于進

* 2014 年春季
  - 向澐, 林家宏, 呂科進, 趙愷文, 阮志偉, 陳建霖


Real Time 的定義
==============================

* Soft Real Time
  
  根據 Scheduling 的結果執行任務。

* 95% Hard Real Time
  
  系統至少有 95% 在指定時間內完成任務。 

* 100% Hard Real Time
  
  系統能 100% 在指定時間內完成任務。 

* Safe Real Time
  
  系統能 100% 在指定時間內完成任務。

Real time on Linux
==================

* PREEMPT_RT (in-kernel; single kernel)

  修改原本的 GNU/Linux 核心 (vanilla kernel),以減少non-preemptible section的方式,使其逐步改善 real-time 能力。

* dual kernel (如 RTLinux, RTAI, Xenomai)

  運作一個 real-time 核心,然後將修改過的 GNU/Linux 核心程式碼視為該 real-time 核心的 idle task。

  在xenomai中,dual kernel 就是 Xenomai 的 Nucleus / Cobalt Core 和 Linux kernel。Xenomai 改變整個系統架構,讓 ipipe -> xenomai scheduler 來預先處理 real-time task,而 Linux 則拉到上層成為一個task。這樣可以避免 Linux 因為龐大的架構而影響處理 real-time 的時間。

Xenomai系統架構
===============

.. image:: /embedded/xenomai/xenomai_arch.jpg

Xenomai是一個linux kernel的patch
藉由在底層增加一個架構
負責硬體與接收interrupt 並將interrupt 傳給上層的OS(這邊稱為domain)
 
這個底層的架構是Adeos 是另一個open source的project


在api呼叫上可以看到不同層級的抽象化

ipipe_XXX -> rthal_XXX -> xnXXX

負責傳送interrupt的程式稱為ipipe
示意圖 : 

.. image:: /embedded/xenomai/adeos.jpg

可以找到ipipe_raise_irq()將interrupt推到pipeline

在ipipe上每個domain都有自己的優先度
高優先度的domain會先接收到interrupt
高優先度的domain的thread 可以preempt低優先度domain的thread

Xenomai 3
+++++++++++

xenomai3有兩種configuration:

* Cobalt: 採用dual kernel架構,是xenomai 2的延伸

* Mercury: 使用單kernel形式,在linux kernel上提供xenomai api,由於本身依賴linux,一般來說會以PREEMPT_RT提供real-time services

**Xenomai 3 dual kernal configuration : Cobalt**

.. image:: /embedded/xenomai/cobalt.png

多一個 priority 比 linux 還高叫 cobalt  的 core 去處理 real-time 的事情,提供不同的 real-time API 給不同的 applications 使用。並且利用Optimistic interrupt protection 機制減少 changing the interrupt mask,一般的機制在每次進入critical section時都要interrupt mask,而Optimistic interrupt protection可以不用。而real-time 在意的 "deadline",實際上就是探討 latency (latency 越大,系統越難在時限內完成完成高優先權任務,自然即時能力就越差),而 latency 很大的來源則是 interrupt handling。

**Xenomai 3 single kernel configuration :Mercury**

.. image:: /embedded/xenomai/mercury.png

運用本機的 linux core 在 PREEMPT_RT之上達到 real-time 的事情,這裡不是強制的,看 applications 對反應時間和 maximum jitter 的要求,有些甚至會作到某種程度 deadline 的忽略。

Adeos / iPipe
++++++++++++++

主要負責處理irq 與 high resolution timer, ipipe的工作很簡單 就是設定timer並將interrupt往上丟

* Adeos feature

  - event pipeline : 

    利用pipeline的方式將不同domain的interrupt或system call往上丟

  - Optimistic interrupt protection :

    當同一個domain在處理interrupt時,有跟她相同domain的interrupt要進來時,會將它進pending狀態,等到所有pending interrupt完成時,才會處理下個domain的interrupt。但更高優先權domain的interrupt會preempt較低優先權domain的interrupt。

  - System­ event propagation :

    system event(ex : page fault handle) 傳遞方式不同於interrupt,基本上是不可被stall的。

* realtime support to threads running in the secondary domain

  - Common priority scheme : 

    當xenomai task migrates to linux domain時,linux domain會繼承xenomai task的priority。

    .. image:: /embedded/xenomai/pirority_scheme.png

  - Predictability of program execution times :

    當xenomai threads 進入linux(secondary) domain時,不可被linux domain interrupt preempt掉,也不能被其他low priority activity at linux kernel preempt掉。通常最簡單實作方式就是加一個interrupt shield domain。

    .. image:: /embedded/xenomai/interrupt_shield.png

  - Fine-grained Linux kernel :

    In order to get the best from secondary domain,we need the Linux kernel to exhibit the shortest possible nonpreemptible section, so that rescheduling opportunities are taken as soon as possible after a Xenomai thread running in the secondary domain becomes ready to run.

  - Priority inversion management :

    Both the real-time nucleus and the Linux kernel should handle the case where a high priority thread is kept from running because a low priority one holds a contented resourse for a possibly unbounded amount of time.

* 相關檔案︰

  - gic.c : 
       
    Generic Interrupt Controller, Interrupt prioritization and distribution to each CPU interface. This is known as the Distributor. Priority masking and preemption handling for each CPU. This is known as the CPU Interface.

  - it8152.c:IRQ相關

  - timer-sp.c:dual timer module(sp804)

  - vic.c:

    The VIC provides a software interface to the interrupt system. In a system with an interrupt controller, software must determine the source that is requesting service and where its service routine is loaded. A VIC does both of these in hardware.

    功能為提供一個programable的介面讓使用者設定

  - ipipe-tsc.c:設定精準度(刻度)

  - ipipe/compat.c:interrupt

  - sched/clock.c:取得cpu_clock 解析度為nanosecond,開機後從0開始上數

.. image:: /embedded/xenomai/cpu_distribute.jpg

GIC大約是上圖的distributor的位置

VIC則是CPU interface的位置

但raspberry pi只有一顆CPU所以不會有SMP與 CPU affinity設定的問題


HAL
++++++++++++

檔案位置在 : xenomai-head/ksrc/arch/arm/hal.c (xenomai 2.6)

Hardware Abstract Layer:process  透過HAL呼叫ipipe的服務。這一層主要是包裝ipipe 與底層資訊 讓nucleus可以不用看到硬體資訊。

Nucleus / Cobalt
++++++++++++++++

檔案位置在 : xenomai-head/ksrc/nucleus (xenomai 2.6) ; xenomai-head/kernel/cobalt (xenomai 3)

Xenomai的kernel, 包含schedule、timer、synch、thread、lock等等一般該有的RTOS功能,負責real-time tasks的執行。

Scheduler
++++++++++++

優先處理realtime task ,linux也被視為其中一個thread,本身也有scheduler,但須等到沒有real-time task時(idle state),才會執行linux thread

.. image:: /embedded/xenomai/xenomai_sched.jpg

就檔案有五個關於sched.c應該有四種不同的schedule方式

sched-idle.c :是專門處理idle狀態給linux schedule使用

sched-rt.c : 給real-time scheduler使用(FIFO+RR)

sched-sporadic.c : POSIX SCHED_SPORADIC scheduling class. 

sched-tp.c : Temporal partitioning (typical of IMA systems)

sched.c : 應該是負責四個schedule方式的檔案


Skins
++++++++++++

檔案位置在xenomai-head/ksrc/skins (xenomai 2.6)

呼叫xenomai的界面, 有native rtdm posix psos+ uitron vrtx vxworks等。


與PREEMPT_RT的差異
===============================

* Linux kernel preemption model 組態 (realtime程度 ↑ => latency↓ but throughput↓)
  - PREEMPT_NONE

    著重fairness和throughput,process在執行system call時無法被preempt。

  - PREEMPT_VOLUNTARY(DESKTOP)

    允許一個低優先權的 process把自己 preempt掉(即便該 process正在 kernel mode 執行著一個系統呼叫)

  - PREEMPT(Low-Latency Desktop)
  - PREEMPT_RT

    著重determinism,對即時系統而言,作業系統的「可決定性」比 throughput 更為重要,使用固定優先權的 preemptive 排程策略 (POSIX SCHED_FIFO 與 SCHED_RR)。


* PREEMPT_RT 機制
  - Preemptible critical sections
  - Preemptible interrupt handlers
  - Preemptible "interrupt disable" code sequences
  - Priority inheritance for in-kernel spinlocks and semaphores
  - Deferred operations
  - Latency-reduction measures
  - Execute all activities (including IRQ) in “schedulable/thread” context

  原本無法preempt的地方讓他可以preemt,讓spinlock 區塊在區分成可以preempt的地方跟不能preempt的地方,將IRQ handler移到thread中執行。

  Priority inheritance 是讓握有spinlock 或 semaphore的process可以暫時的提高優先權 讓他可以盡快做完critical section釋放spinlock或semaphore

  高Priority的 process才有辦法繼續執行。

* PREEMPT_RT 與 xenomai的差異

  RT_PREEMPT是基於linux架構去改進 讓更多地方能preempt 達到real-time的能力

  Xenomai則是改變整個系統架構 新增一個scheduler與IRQ管理的機制

  讓處理real-time task流程簡化到只剩ipipe->scheduler 就能執行

  不會因linux龐大的架構影響到real-time task的處理時間


建立 Xenomai 環境
==============================

Xenomai 2.6 on raspberry pi
++++++++++++++++++++++++++++

* 下載 Raspbian: http://www.raspberrypi.org/downloads/
  - 將raspbian的img檔燒進sd card上

.. code-block:: c

    sudo dd if=<raspbian image file> of=/dev/mmcblk0 bs=4M

  - ``if`` 為 input file, ``of`` 為 output file, ``bs``為 block size
  - 燒錄詳細介紹: http://life-of-raspberrypi.blogspot.tw/

* Install Cross complier

.. code-block:: c

    cd <working dir>
    wget https://github.com/raspberrypi/tools/archive/master.tar.gz
    tar xzfv master.tar.gz

* Download kernel
 
.. code-block:: c

   git clone -b rpi-3.8.y --depth 1 git://github.com/raspberrypi/linux.git linux-rpi-3.8.y

* Download Xenomai 

.. code-block:: c

   git clone git://git.xenomai.org/xenomai-head.git xenomai-head

* Download minimal config 

.. code-block:: c

   wget https://www.dropbox.com/s/dcju74md5sz45at/rpi_xenomai_config

* Apply ipipe core pre-patch

.. code-block:: c

   cd linux-rpi-3.8.y
   patch -Np1 < ../xenomai-head/ksrc/arch/arm/patches/raspberry/ipipe-core-3.8.13-raspberry-pre-2.patch

* Apply Xenomai ipipe core patch 

.. code-block:: c

   cd <working dir>
   ./xenomai-head/scripts/prepare-kernel.sh --arch=arm --linux=linux-rpi-3.8.y --adeos=xenomai-head/ksrc/arch/arm/patches/ipipe-core-3.8.13-arm-4.patch

* Apply ipipe core post-patch 

.. code-block:: c

   cd linux-rpi-3.8.y
   patch -Np1 < ../xenomai-head/ksrc/arch/arm/patches/raspberry/ipipe-core-3.8.13-raspberry-post-2.patch

* Create build directory 

.. code-block:: c

   cd <working dir>
   mkdir linux-rpi-3.8.y/build

* Configure kernel

.. code-block:: c

   cp rpi_xenomai_config linux-rpi-3.8.y/build/.config
   cd linux-rpi-3.8.y
   make mrproper
   make ARCH=arm O=build oldconfig

* Compile Linux Kernel (此步驟耗時長,建議用make -j平行加速)

在作此步驟之前須export library

.. code-block:: c

  export PATH=<working dir(full path)>/tools-master/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/:$PATH

.. code-block:: c

   make ARCH=arm O=build CROSS_COMPILE=<working dir(full path)>/tools-master/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/arm-bcm2708hardfp-linux-gnueabi-
   
* Install modules

.. code-block:: c

  make ARCH=arm O=build INSTALL_MOD_PATH=dist modules_install

* Install headers

.. code-block:: c

    make ARCH=arm O=build INSTALL_HDR_PATH=dist headers_install
    find build/dist/include \( -name .install -o -name ..install.cmd \) -delete

* 編譯好的kernelImage(<working dir>/linux-rpi-3.8.y/build/arch/arm/boot/Image),移到SD卡的 ``/boot/`` 路徑下並更改名稱為kernel.img

* 將``linux-rpi-3.8.y/build/dist``中的Module,移到SD卡中的``/lib/modules``

* Compile Xenomai

.. code-block:: c
    
  cd xenomai-head
  export PATH=<working dir(full path)>/tools-master/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi/bin/:$PATH
  ./configure --host=arm-bcm2708hardfp-linux-gnueabi
  cd src
  mkdir dist
  make install DESTDIR=`pwd`/dist

* dist中會出現``usr/xenomai``, 將這個資料夾移到sd卡中 ``/usr/``

* 用 minicom 連進 raspberry pi 中執行以下動作

.. code-block:: c

  export PATH=/usr/xenomai/bin:$PATH  
  export LD_LIBRARY_PATH=/usr/xenomai/lib
  sudo modprobe xeno_posix

Xenomai 3 on raspberry pi
++++++++++++++++++++++++++

* 取得 xenomai 3.0-rc、rpi-linux 3.16.y、ipipe 3.18.12 arm patch、toolchain

.. code-block:: c

  git clone -b rpi-3.18.y git://github.com/raspberrypi/linux.git rpi-linux
  git clone -b v3.0-rc4 git://git.xenomai.org/xenomai-3.git xenomai-3
  wget http://download.gna.org/adeos/patches/v3.x/arm/ipipe-core-3.18.12-arm-1.patch
  git clone https://github.com/raspberrypi/tools.git --depth=1

* 由此處取得rpi post patch

`post patch<https://drive.google.com/file/d/0B6GwSked9fL9RnU3b3pSajFZbUk/view>`_

*上 ipipe patch

.. code-block:: c

  cd rpi-linux
  ../xenomai-3/scripts/prepare-kernel.sh --arch=arm --ipipe=<你的patch位置>/ipipe-core-3.18.12-arm-1.patch   --linux=.
  patch -Np1 < 3.18.12-xenomai3-temp.patch

* configure kernel(使用rpi提供的default)

.. code-block:: c

  cd rpi-linux
  export CCPI=($working_dir)/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-
  mkdir build
  make mrproper
  make ARCH=arm O=build CROSS_COMPILE=$CCPI bcmrpi_defconfig
  make ARCH=arm O=build CROSS_COMPILE=$CCPI menuconfig

若patch正確,會在設定表裡看到xenomai相關選項。

vim build/.config,找到CONFIG_CPU_FREQ , CONFIG_CPU_IDLE, CONFIG_KGDB, CONFIG_CONTEXT_TRACKING_FORCE(若有的話)設為n。

* compile kernel

.. code-block:: c

  make -j 5 ARCH=arm O=build CROSS_COMPILE=$CCPI

* 安裝kernel modules

.. code-block:: c

  make ARCH=arm O=build INSTALL_MOD_PATH=dist modules_install

* 安裝headers

.. code-block:: c

  make ARCH=arm O=build INSTALL_HDR_PATH=dist headers_install

* xenomai

.. code-block:: c

  cd ../xenomai-3
  mkdir dist
  export PATH=/home/erickitten/workspace/xenomai/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/:$PATH

* 設定bootstrap

.. code-block:: c

  ./scripts/bootstrap --with-core=cobalt –enable-debug=partial

* configure

.. code-block:: c

  ./configure CFLAGS="-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard" LDFLAGS="-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard" --host=arm-linux-gnueabihf --with-core=cobalt

* install

.. code-block:: c

  make DESTDIR=`pwd`/dist install

* 複製kernel

將linux-rpi/build/arch/arm/boot/Image複製到SD卡的/boot (partition),並改名成kernel.img。或使用config.txt,以kernel=<filename>指定名稱。

* 移動module / patches (權限問題,需使用sudo)

.. code-block:: c

  cd ..
  sudo cp -r rpi-linux/build/dist/lib/modules $(sdcard)/lib
  sudo cp -r xenomai-3/dist/usr/xenomai $(sdcard)/usr

Beaglebone Black 環境設置
++++++++++++++++++++++++++++

* 下載Kernel Source & 編譯

.. code-block:: c

  cd ti-linux-kernel-dev
  git checkout origin/ti-linux-3.14.y -b tmp
  ./build_kernel.sh

* 預設username/password

.. code-block:: c

  username:ubuntu
  password:temppwd

* 設定cross compile

在Kernel Configuration中選擇General setup,輸入

.. code-block:: c

  CROSS_COMPILE=arm-linux-gnueabi-

* 取得image,並且驗證 (注意:注意md5碼須相同)

.. code-block:: c

  wget https://rcn-ee.com/rootfs/2015-04-09/ubuntu-14.04.2-console-armhf-2015-04-09.tar.xz
  md5sum ubuntu-14.04.2-console-armhf-2015-04-09.tar.xz
  be1f3008aaa32d88c842897e324bcdf8  ubuntu-14.04.2-console-armhf-2015-04-09.tar.xz

* 解壓縮image

.. code-block:: c

  tar xf ubuntu-14.04.2-console-armhf-2015-04-09.tar.xz
  cd ubuntu-14.04.2-console-armhf-2015-04-09

* 檢查SD Card位址

.. code-block:: c

  sudo ./setup_sdcard.sh --probe-mmc

* 假設現在位址是 "/dev/mmcblk0"

注意,ubuntu 一定要加 bash,因為 ubuntu 沒設定 default 為 bash

.. code-block:: c

  sudo bash setup_sdcard.sh --mmc /dev/mmcblk0 --dtb beaglebone

* Advanced: Build Image 

.. code-block:: c

  git clone https://github.com/RobertCNelson/omap-image-builder.git
  cd omap-image-builder
  git checkout v2015.04 -b tmp

* Stable

.. code-block:: c

  ./RootStock-NG.sh -c rcn-ee_console_ubuntu_stable_armhf

* 完成上述步驟後即可上電,按 boot select button 讓它從 sd card 開機,開機後進入登入畫面,帳號/密碼寫在上面

* 登入後查詢版本

.. code-block:: c

  sudo su
  uname -a

Xenomai 2.6 on Beaglebone
++++++++++++++++++++++++++++

使用舊版linux(Angstrom)來做測試,核心編譯步驟參考上面步驟 或 http://elinux.org/EBC_Installing_Kernel_Source

Kernel裝好後,測試可以成功從 SD 卡開機,接著更換換 Kernel

參考資料 http://elinux.org/EBC_Xenomai 與 http://emplearn.blogspot.tw/search?q=xenomai

* 設定路徑

.. code-block:: c

  mkdir bbb
  cd bbb
  export BBB=$(pwd)

* 取得xenomai-2.6.4 (與 beaglebone-kernel 同一層目錄)

.. code-block:: c

  wget http://download.gna.org/xenomai/stable/latest/xenomai-2.6.4.tar.bz2

* 檢查版本

.. code-block:: c

  cd bb-kernel/KERNEL
  uname -a
  git tags | sort | less

* 選擇跟所編譯 Kernel 最接近版本,在此為 3.8.13

.. code-block:: c

  git checkout 3.8.13-bone67 -b xenomai

* Patch the kernel

.. code-block:: c

  cd bb-kernel/KERNEL
  patch -p1 < ../../xenomai-2.6.4/ksrc/arch/arm/patches/beaglebone/ipipe-core-3.8.13-beaglebone-pre.patch
  patch -p1 < ../../xenomai-2.6.4/ksrc/arch/arm/patches/ipipe-core-3.8.13-arm-4.patch
  patch -p1 < ../../xenomai-2.6.4/ksrc/arch/arm/patches/beaglebone/ipipe-core-3.8.13-beaglebone-post.patch

* Prepare the kernel

.. code-block:: c

  cd ../../xenomai-2.6.4/scripts
  ./prepare-kernel.sh --arch=arm --linux=../../bb-kernel/KERNEL/

* Prepare the kernel

.. code-block:: c

  cd ../../bb-kernel

* 確認已經關掉AUTO_BUILD

.. code-block:: c

  vi system.sh

* 確認最後一行為

.. code-block:: c

  #AUTO_BUILD=1

* 關掉 CPU Power Management ---> CPU Frequency scaling中的 [ ] CPU Frequency scaling.

* 將Real-time sub-system ---> Drivers ---> Testing drivers中的選項全部打開

.. code-block:: c

 cd ../../bb-kernel/KERNEL
 make ARCH=arm menuconfig

* 再重新編譯

.. code-block:: c

  make -j8 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LOADADDR=0x80008000 uImage dtbs modules

* 編譯完後將 SD 卡內的 Kernel 換掉

.. code-block:: c

  mkdir sd

* 掛載 sd 卡

* 確認掛載的目錄內一定要有這些路徑

.. code-block:: c

  sudo mount /dev/mmcblk0p1 sd
  sudo cp beagle-kernel/kernel/arch/arm/boot/uImage sd/boot/uImage-3.8.13
  cd beagle-kernel/kernel
  sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- INSTALL_MOD_PATH=$BBB/sd modules_install
  cd -
  mkdir sd/home/root/xeno_drivers
  cp beagle-kernel/kernel/drivers/xenomai/testing/*.ko sd/home/root/xeno_drivers/
  cp -r xenomai-2.6.3 sd/home/root
  ./configure --target=arm-linux-gnueabi
  mkdir -p dist; make DESTDIR=`pwd`/dist install

* 將編譯後dist內的內容複製放到beaglebone上

.. code-block:: c

  sudo umount sd

* 測試,掛載driver

.. code-block:: c

  cd ~/xeno_drivers
  insmod xeno_klat.ko
  /usr/xenomai/bin/cyclictest -p 90





觀察與分析
=========

.. code-block:: prettyprint

     pi@raspberrypi:~$ cat /proc/xenomai/stat
     CPU    PID        MSW                  CSW           PF         STAT            %CPU    NAME
     0      0            0                  206           0          00500080        100.0   ROOT
     0      0            0                  2688553       0          00000000        0.0     IRQ3: [timer]

* CPU : 目前這個tread是使用哪個CPU在運行,而rpi是單核心CPU,故顯示皆為0
* MSW : Mode SWitches, This value should only increase over time for threads that are expected to interact with Linux services.
  - 當process從primary mode轉成secondary mode或是secondary mode轉成primary mode時,將會紀錄一次的轉換。

  - cyclictest的RT task因為會執行到memset,所以會從xenomai schedule跳到linux schedule,MSW+1,而執行完memset後將在跳回xenomai schedule,故再+1

* CSW : Number of Context SWitches (or IRQ hits for the particular CPU)
* PF : Number of Page Faults (should stop increasing as soon as mlockall is in effect)
* STAT : A bitfield describing the internal state of the thread. Bit values are defined in include/nucleus/thread.h (See status and mode bits). The STAT field from /proc/xenomai/sched gives a 1-letter-per-bit symbolic translation of a the most significant subset of those bits.
* %CPU : CPU share of the thread (or IRQ handler) since the last retrieval of the statistics.
* NAME : Name of the thread (or IRQ number and registered driver). Can be set, e.g., with the (non portable) POSIX-API-function pthread_set_name_np. See API documentation of the RTOS skin in question.

.. code-block:: prettyprint

     pi@raspberrypi:~$ sudo /usr/xenomai/bin/cyclictest >/dev/null 2>/dev/null &
     [1] 2253

.. code-block:: prettyprint

     pi@raspberrypi:~$ ps aux | grep -i "cy"
     root      2253  0.5  0.3   4580   1464  ?        S    03:34   0:00   sudo /usr/xenomai/bin/cyclictest
     root      2254  2.7  0.4   2340   2132  ?        SLl  03:34   0:00   /usr/xenomai/bin/cyclictest
     pi        2259  0.0  0.1   3540   820   ttyAMA0  S+   03:34   0:00   grep --color=auto -i cy

.. code-block:: prettyprint

     pi@raspberrypi:~$ cat /proc/xenomai/stat
     CPU    PID        MSW                CSW              PF        STAT        %CPU    NAME
     0      0          0                  255              0         00500080    100.0   ROOT
     0      2254       1                  1                0         00b00380    0.0     cyclictest
     0      2256       2                  48               0         00300184    0.0     cyclictest
     0      0          0                  2913946          0         00000000    0.0     IRQ3: [timer]

.. code-block:: prettyprint

     pi@raspberrypi:~$ watch -n 1 cat /proc/xenomai/stat
     Every 1.0s: cat /proc/xenomai/stat                      Wed Jan  8 03:38:43 2014

     CPU    PID        MSW                CSW           PF        STAT         %CPU     NAME
     0      0          0                  442           0         00500080     99.9     ROOT
     0      2254       1                  1             0         00b00380     0.0      cyclictest
     0      2256       2                  235           0         00300184     0.0      cyclictest
     0      0          0                  2953543       0         00000000     0.1      IRQ3: [timer]

在這邊可以看到cyclictest有兩個pid,因為/usr/xenomai/bin/cyclictest它會先創一個thread,並讓這個thread跑nanosleep,所以會有兩個process。接著看向CSW,pid 2254的cyclictest, 他的CSW只有1。pid 2256的卻有235,這是因為2256是一個xenomai realtime task,而 2254是一個 linux的process,所以2256會優先執行,直到realtime task都做完才會換low priority的linux domain process取得CPU,因此2254的CSW值才會是1而沒有增加。

.. code-block:: prettyprint

     pi@raspberrypi:~$ sudo kill 2254

     pi@raspberrypi:~$ ps aux | grep -i "cy"
     pi        2324  0.0  0.1   3540   820 ttyAMA0  R+   03:46   0:00 grep --color=auto -i cy
     [1]+  Done                    sudo /usr/xenomai/bin/cyclictest > /dev/null 2> /dev/null

     pi@raspberrypi:~$ sudo /usr/xenomai/bin/cyclictest -p FIFO >/dev/null 2>/dev/null &

* 在我們了解MSW時,嘗試了在-p後面加上了文字(如:FIFO、RR……)  

* 發現MSV的值開始往上增加,也發現一開始對於MSW的定義理解錯誤

.. code-block:: prettyprint

      CPU    PID        MSW                CSW                PF         STAT          %CPU     NAME
      0      0          0                  75266              0          00500080      99.9     ROOT
      0      2978       1                  1                  0          00b00380      0.0      cyclictest
      0      2980       2                  26846              0          00300184      0.0      cyclictest
      0      7559       1                  1                  0          00b00380      0.0      cyclictest
      0      7561       66                 130                0          00b00184      0.0      cyclictest
      0      0          0                  11266931           0          00000000      0.1      IRQ3: [timer]

* trace後才了解,這是xenomai在-p的指令上是使用atoi,將輸入的數字轉為int,但並沒有進行偵錯,才導致segment fault,而需跳轉到linux domain進行除錯。

效能表現
=======

* Stock Linux 

.. code-block:: prettyprint

   cyclictest -p 90 - m -c 0 -i 200 -n -h 100 -q -l 1000 >log

.. image:: /embedded/xenomai/001.png

* PREEMPT_RT-patched Linux

.. code-block:: prettyprint

   cyclictest -p 90 - m -c 0 -i 200 -n -h 100 -q -l 1000 >log

.. image:: /embedded/xenomai/preemptRt.png

* Xenomai-patched Linux

.. code-block:: prettyprint

   /usr/xenomai/bin/cyclictest -p 90 - m -c 0 -i 200 -n -v 100 -q -l 100" >log

.. image:: /embedded/xenomai/002.png

Cyclictest 原理
==============

* 概念:設定一個時間間隔->取得現在時間->讓process 睡一個間隔->process醒來後再取一次時間->比對兩次取得的時間差與設定的間隔差距

* pseudocode:

  .. code-block:: prettyprint

    clock_gettime((&now))
    next = now + par->interval
    while (!shutdown) {
        clock_nanosleep((&next))
        clock_gettime((&now))
        diff = calcdiff(now, next)
        # update stat-> min, max, total latency, cycles
        # update the histogram data
        next += interval
    }

* 造成時間差的原因

  - timer精準度
  - IRQ latency
  - IRQ handler duration
  - scheduler latency
  - scheduler duration


* Cyclictest 實作流程

  1.cyclictest建立一個timerthread, 它一個 realtime 的 thread

  2.timerthread會重複的執行取第一次時間 nanosleep(interval) 取第二次時間 比對兩次時間差與interval的差異

  3.最後將結果輸出在terminal


* Clock_nanosleep 的 timer

  clock_nanosleep 使用的timer 是 high resolution timer(HRT) ,讓睡眠時間可以更精確,達到nanosecond的精準度(但還是要看硬體能提供的精準度)

  因為能在更準確得時間讓process醒來並取的nanoscecond單位的時間 所以可以計算到由systick無法計算到的duration + latency


* Clock_nanosleep 實作流程

  1.使用 spinlock (xnlock_get_irqsave) 令 CPU 不接受 Interrupt 

  2.使用 xnpod_suspend_thread 改變目前 thread 的狀態

  3.使用 xntimer_get_timeout_stopped 取得 tick

  4.使用 ticks2ts 轉換時間單位



.. code-block:: prettyprint

     int clock_nanosleep (clockid_t clock_id, int flags, const struct timespec *rqtp, struct timespec *rmtp)
     {
        xnthread_t *cur;
        spl_t s;
        int err = 0;

        if (xnpod_unblockable_p())
            return EPERM;

        if (clock_id != CLOCK_MONOTONIC && clock_id != CLOCK_REALTIME)
            return ENOTSUP;

        if ((unsigned long)rqtp->tv_nsec >= ONE_BILLION)
            return EINVAL;

        if (flags & ~TIMER_ABSTIME)
            return EINVAL;

        cur = xnpod_current_thread();

        xnlock_get_irqsave(&nklock, s); 

        thread_cancellation_point(cur);

        xnpod_suspend_thread(cur, XNDELAY, ts2ticks_ceil(rqtp) + 1,clock_flag(flags, clock_id), NULL);

        thread_cancellation_point(cur);

        if (xnthread_test_info(cur, XNBREAK)) {

            if (flags == 0 && rmtp) {
                xnsticks_t rem;

                rem = xntimer_get_timeout_stopped(&cur->rtimer);
                xnlock_put_irqrestore(&nklock, s); 

                ticks2ts(rmtp, rem > 1 ? rem : 0); 
            } else
                xnlock_put_irqrestore(&nklock, s); 

            return EINTR;
        } 
  
        xnlock_put_irqrestore(&nklock, s);

        return err; 
     }

* `Cyclictest<https://rt.wiki.kernel.org/index.php/Cyclictest>`_
* Test case: POSIX interval timer, Interval 500 micro seconds,. 100000 loops, 100% load.
  - Commandline: cyclictest -t1 -p 80 -i 500 -l 100000

* 使用 PREEMPT LINUX

.. code-block:: prettyprint

    root@raspberrypi:/home/pi# sudo ./cyclictest -t1 -p 80 -i 500 -l 100000
    # /dev/cpu_dma_latency set to 0us
    policy: fifo: loadavg: 0.00 0.01 0.05 1/61 2064          
    T: 0 ( 2063) P:80 I:500 C: 100000 Min:     27 Act:   49 Avg:   42 Max:    1060

* 使用 RT-PREEMPT

.. code-block:: prettyprint

    Linux raspberrypi 3.6.11+ #474 PREEMPT Thu Jun 13 17:14:42 BST 2013 armv6l GNU/Linux
    Min:     22 Act:   31 Avg:   32 Max:     169

* 使用 Xenomai

.. code-block:: prettyprint

    Linux raspberrypi 3.8.13-core+ #1 Thu Feb 27 03:02:16 CST 2014 armv6l GNU/Linux
    Min:      1 Act:    5 Avg:    6 Max:      41

.. code-block:: prettyprint

    root@raspberrypi:/home/pi# /usr/xenomai/bin/cyclictest -t1 -p 80 -i 500 -l 10000 
    0.08 0.06 0.05 1/61 2060          
    T: 0 ( 2060) P:80 I:     500 C:  100000 Min:      -4 Act:      -2 Avg:       0 Max:      30

T:thread

P:priority

I:interval

C:執行cycle數

Min:最小延遲

Act:此次延遲時間

Avg:平均延遲

Max:最大延遲

最重要的是Max值 為了確保realtime 要能知道worst case 

讓開發者可以評估最差的情況可以在多少時間內可以做出回應

Q&A
==============

* Q1:handler duration 與 schedule latency 之間的延遲原因為何?
  - A: 

  .. image:: /embedded/xenomai/tasklatency.jpg


Hackpad
=======
*  討論&紀錄 https://embedded2014.hackpad.com/Xenomai-raspberry-note-XwJtuQn9nkD

*  整理 https://embedded2014.hackpad.com/Xenomai-z2CJPjPLTer

* 2015 hackpad:https://embedded2015.hackpad.com/Xenomai-LydARMEzEWP#:h=Cobalt

參考資料
=======
* https://code.google.com/p/picnc/wiki/RPiXenomaiKernel
* https://code.google.com/p/picnc/wiki/CreateRaspbianLinuxCNC
* http://www.camelsoftware.com/firetail/blog/raspberry-pi/real-time-operating-systems/
* `Quadruped Linux robot feels its way over obstacles<http://linuxgizmos.com/hyq-quadruped-robot-runs-real-time-linux/>`_
* ` Choosing between Xenomai and Linux for real-time applications<https://www.osadl.org/fileadmin/dam/rtlws/12/Brown.pdf>`_
* `Real Time Systems<http://www.slideshare.net/anil_pugalia/real-time-systems>`_
* http://www.cs.ru.nl/lab/xenomai/exercises/
* `背景知識 <https://www.osadl.org/fileadmin/dam/rtlws/12/Brown.pdf>`_
* `應用案例 <http://veter-project.blogspot.tw/search/label/xenomai>`_
* `應用案例II <https://www.youtube.com/watch?v=cNZPRsrwumQ>`_
* `real-time Linux 介紹 <http://class.svuca.edu/~sau/class/CE460/IntroEmbeddedLinuxDesign_Spring2014_Sec_13.ppt>`_
* `Xenomai 專案維護人的介紹 <http://elinux.org/images/7/70/Gerum-elce-09.odp>`_
* `RTOS 定義 <http://open-robotics.com/report/rtos/>`_