Simple Rule makes simple progress

 Test

Get the max from bazel: use a dashboard to track the build status

Build the dashboard server:

$ ./output/bazel build //src/tools/dash:all

Run it locally

$ bazel-bin/src/tools/dash/dash

Then you can stream your build result to this local server:

$ bazel build --use_dash --dash_url http://localhost:8080 lte/base:all

Or deploy it on appengine
$ bazel-bin/src/tools/dash/dash.deploy app-engine-app-id

Then use
http://app-engine-app-id.appspot.com to access it.

Use Namespace to gain special capabilities for testing(without docker)

Starting with kernel 2.2, Linux divides the privileges traditionally associated with superuser into distinct units, known as capabilities, which can be independently enabled and disabled.  Capabilities are a per-thread attribute.

For example, to be able to configure network setting, instead of becoming root user, if the process has CAP_NET_ADMIN capability, the relevant syscall will be permitted by kernel.

More information about capabilities can be found in Linux man page: man 7 CAPABILITIES.


Linux provides the following namespaces

       Namespace   Constant        Isolates
       IPC         CLONE_NEWIPC    System V IPC, POSIX message queues
       Network     CLONE_NEWNET    Network devices, stacks, ports, etc.
       Mount       CLONE_NEWNS     Mount points
       PID         CLONE_NEWPID    Process IDs
       User        CLONE_NEWUSER   User and group IDs
       UTS         CLONE_NEWUTS    Hostname and NIS domain name


User namespaces isolate security-related identifiers and attributes, in particular, user IDs and group IDs (see credentials(7)), the root directory, keys (see keyctl(2)), and capabilities (see capabilities(7)).

When a new user namespace is created(either using clone or unshare), it starts with *a complete/full set of capabilities*.

But that's good enough for us to run some test program inside. Inspired by the namespace-sandbox.c tool from bazel, I wrote a simpler version new-network-namespace.c that just creates an empty user namespace with network namespace and launch a given program inside the namespace. Inside that program, the user can perform ifconfig/iptables operation without being the real root, and without
worrying about break the real system by accident.

For example:

$ ./new-network-namespace /bin/bash
root@myhost:~# ifconfig -a
lo        Link encap:Local Loopback
          LOOPBACK  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Please give it a try, :)

Build DPDK with bazel

bazel is the open sourced version of Google build tool, and it has proved itself is a nice solution for monolithic server side development. DPDK is a nice solution for user mode packet processing. In our company, we use both to build our products and it greatly reduced the hurdle of rolling out any change because we put the whole DPDK source code into our source tree and the build process can easily produce final binaries at any point(i.e. we can put a fix to DPDK code and the build process will always produce up-to-date binary based on the change).

We pushed all our changes to github in "bazel" branch in case any one wants to try it. At the moment, due to the complexity of setting up bazel BUILD files to handle different compilation settings, we are not actively pursuing to integrate these changes back to upstream. But I'll be happy to answer any question and provide help in case anyone wants to do so.

Quickly turn gentoo daily snapshot into docker image

Download the snapshot from gentoo mirror:

Then
$ bunzip2 stage3-amd64-20141120.tar.bz2 -c | docker import - gentoo-amd64
7dbd254474e511597f160342bf8d828406f52467a62061b29ea0b3009b806b05

That's it!

Write zookeeper client log using google glog

I don't like the fact every other piece of log in my program is happily written by google glog but zookeeper is messing stderr by writing the log there. Besides, I like the format of glog better so I made a quick change to ask zookeeper c clien to write log using glog, here is a quick patch:

diff --git a/third-party/zookeeper/src/c/include/zookeeper_log.h b/third-party/zookeeper/src/c/include/zookeeper_log.h
index e5917cb..6519587 100644
--- a/third-party/zookeeper/src/c/include/zookeeper_log.h
+++ b/third-party/zookeeper/src/c/include/zookeeper_log.h
@@ -28,6 +28,24 @@ extern "C" {
 extern ZOOAPI ZooLogLevel logLevel;
 #define LOGSTREAM getLogStream()

+#define ZOOKEEPER_GLOG
+#ifdef ZOOKEEPER_GLOG
+ZOOAPI void zk_glog_message(int curLevel, int line, const char* funcName,
+                            const char* message);
+/* We can't include glog/log_severity.h because it's in C++ style, so
+ * we hard code the corresponding log level here */
+#define LOG_ERROR(x) if(logLevel>=ZOO_LOG_LEVEL_ERROR) \
+    zk_glog_message(2, __LINE__, __FILE__, format_log_message x)
+#define LOG_WARN(x) if(logLevel>=ZOO_LOG_LEVEL_WARN) \
+    zk_glog_message(1, __LINE__, __FILE__, format_log_message x)
+#define LOG_INFO(x) if(logLevel>=ZOO_LOG_LEVEL_INFO) \
+    zk_glog_message(0, __LINE__, __FILE__, format_log_message x)
+#define LOG_DEBUG(x) if(logLevel==ZOO_LOG_LEVEL_DEBUG) \
+    zk_glog_message(0, __LINE__, __FILE__, format_log_message x)
+#else
+
 #define LOG_ERROR(x) if(logLevel>=ZOO_LOG_LEVEL_ERROR) \
     log_message(ZOO_LOG_LEVEL_ERROR,__LINE__,__func__,format_log_message x)
 #define LOG_WARN(x) if(logLevel>=ZOO_LOG_LEVEL_WARN) \
@@ -36,6 +54,7 @@ extern ZOOAPI ZooLogLevel logLevel;
     log_message(ZOO_LOG_LEVEL_INFO,__LINE__,__func__,format_log_message x)
 #define LOG_DEBUG(x) if(logLevel==ZOO_LOG_LEVEL_DEBUG) \
     log_message(ZOO_LOG_LEVEL_DEBUG,__LINE__,__func__,format_log_message x)
+#endif

 ZOOAPI void log_message(ZooLogLevel curLevel, int line,const char* funcName,
     const char* message);
diff --git a/third-party/zookeeper/src/c/src/zk_glog.cc b/third-party/zookeeper/src/c/src/zk_glog.cc
new file mode 100644
index 0000000..ad2bdb3
--- /dev/null
+++ b/third-party/zookeeper/src/c/src/zk_glog.cc
@@ -0,0 +1,10 @@
+#include "third-party/google/glog/logging.h"
+#include "zookeeper_log.h"
+
+extern "C" {
+void zk_glog_message(int curLevel, int line, const char* filename,
+                     const char* message) {
+  LogMessage(filename, line, curLevel).stream() << message;
+}
+}

Run X11 application inside docker without VNC or SSH

I'm so annoyed that people would think of using ssh forwarding or VNC to run a X11 application inside docker. Docker is just a special chroot environment so there must be a more efficient way to setup the communication tunnel required by X11, and here it is:

docker run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY

/tmp/.X11-unix contains the unix domain socket file used by your running X11 application, but mounting it inside docker, the X11 app inside docker will  happily access it.

Evaluate performance bottleneck with perf

Perf tool is a profiler tool for Linux kernel 2.6+ that uses performance counter to profile the performance bottleneck of a program in both userspace and kernel space.

This tutorial is good example on how to use it.

To install it on ubuntu, sudo apt-get install linux-tools

A quick example on how to find the bottleneck of a program.

1. Run the program as usual to collect performance record, for example "
   perf record iperf -c 192.168.1.1 -d", This command will ask perf to execute command "iperf -c 192.168.1.1 -d" and collect the performance number in "perf.data" file.
2. View the performance record: perf record, this will show you how much time is spent in a userspace function or kernel function. In my example, it looks like this:
   

    31.95%  iperf  [kernel.kallsyms]        [k] md5_transform                                                                                                                        
    16.99%  iperf  [aesni_intel]            [k] _aesni_enc1                                                                                                                          
     3.14%  iperf  [kernel.kallsyms]        [k] do_csum                                                                                                                              
     2.50%  iperf  [kernel.kallsyms]        [k] memcpy                                                                                                                               
     2.16%  iperf  [kernel.kallsyms]        [k] __ticket_spin_lock                                                                                                                   
     1.78%  iperf  [aesni_intel]            [k] _aesni_dec4                                                                                                                          
     1.10%  iperf  [kernel.kallsyms]        [k] nf_iterate                                                                                                                           
     0.97%  iperf  [nf_conntrack]           [k] ____nf_conntrack_find                                                                                                                
     0.91%  iperf  [kernel.kallsyms]        [k] __slab_free                                                                                                                          
     0.81%  iperf  [kernel.kallsyms]        [k] skb_release_data                                                                                                                     
     0.80%  iperf  [kernel.kallsyms]        [k] fib_table_lookup                                                                                                                     
     0.80%  iperf  [kernel.kallsyms]        [k] memset                                                                                                                               
     0.80%  iperf  [kernel.kallsyms]        [k] __copy_user_nocache                                                                                                                  
     0.77%  iperf  [cxgb4]                  [k] process_responses                                                                                                                    
     0.77%  iperf  [ip_tables]              [k] ipt_do_table                                                                                                                         
     0.69%  iperf  [nf_conntrack]           [k] __nf_conntrack_find_get                                                                                                              
     0.68%  iperf  [kernel.kallsyms]        [k] md5_update                                                                                                                           
     0.67%  iperf  [nf_conntrack]           [k] nf_conntrack_in                                                                                                                      
     0.56%  iperf  [nf_conntrack]           [k] hash_conntrack_raw                                                                                                                   
     0.52%  iperf  [kernel.kallsyms]        [k] dst_release                         
   

   
   This is because I'm running tcp benchmark using iperf with ipsec turned on in ESP and AH mode, so a lot of cpu time is spent on md5(for checksum) and aes(for encryption). Now that we know where the performance bottlenecks are, it's time to spend time to fix them.

openwrt now supports WD N750

Felix Kaechele submitted a bunch of patches and it's finally fully supported.
http://wiki.openwrt.org/toh/wd/n750 has a precompiled firmware.

Enable Telnetd on WD My Net N750 Router

My Net N750 Router is a cheap yet crappy router with dual band(2.4G and 5G). The good part is WD provides its source code to download.

After exploring its source code, I found it provides telnet support.

First, go to http://router_ip/telnet.php to enable it.

Then telnet router_ip

The username and password are hardcoded in the firmware: Alphanetworks/wrgnd13_wd_av

So use it at your own risk!

Work around ipsec setkey error "The result of line 2: (null).”

In some Ipsec old howto, AH and ESP are specified through two settings when configuring ipsec SA manually:

add 10.0.0.216 10.0.0.11 ah 24500 -A hmac-md5 "1234567890123456";          
add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012";

spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
   esp/transport//require
   ah/transport//require;

It results error message like “The result of line 2: (null).” when running with “setkey” with 3.7.10 linux kernel. The solution is to combine merge hmac setting into ESP, like this:

add 10.0.0.216 10.0.0.11 esp 24501 -E 3des-cbc "123456789012123456789012"
  -A hmac-md5 "1234567890123456";          

spdadd 10.0.0.216 10.0.0.11 any -P out ipsec
   esp/transport//require
   ah/transport//require;

Understand the overhead of JNI

JNI[1][2] allows java code to call some native code written in C/C++. Inside native code implementation, it is possible to make other function calls to access the data inside JVM, like to access a class’ field, calling another java method, etc. The JNI’s overhead comes from 3 parts:

• Native code prevents the optimization that JVM can make
• Setup the environment to start a jni call
• Data copy between JVM and native code, and indirection to access some fields and methods from native code.

JNI hurts JVM Optimization

The JVM can't inline the native method, no matter how simple it is.

The JVM doesn't know enough about the method to make optimisations that it could make when compiling a regular Java method (for example, it has to assume that all of the parameters passed in are always used);

The JVM can't make other optimization that it could make if it were dynamically compiling the code (e.g. compiling a constant parameter is a constant operand to a machine instruction rather than placing it on the stack and reading it off again);

The pure cost to make a JNI call

In order to make the native call into the DLL or library, the JVM may have to perform extra work, such as rearranging items on the stack.

Suppose we have a dummy native function which does nothing, on a 32bit JVM(openjdk 6) it takes about 10ns to make a single call on a testing machine.

private static native void noopJni();

extern "C" JNIEXPORT void JNICALL
Java_com_foo_test_JniPerfTest_noopJni(
JNIEnv* env, jclass) {
}

Interaction between JVM and native code

If the native code implementation just needs to access the data passed as parameters and returns value, then there is no additional cost involved. But in practice, the native has to do more interactions between jvm, like to access some data from JVM, calling some other Java method, etc. Take the following native code for example, which computes the sum of an integer array:

private static native int sum(int[] src);

extern "C" JNIEXPORT jint JNICALL
Java_com_foo_test_JniPerfTest_sum(
JNIEnv* env, jclass, jintArray src) {
  const jint size = env->GetArrayLength(src);
  jint* data = env->GetIntArrayElements(src, 0);
  jint sum = 0;
  for (int i = 0; i < size; ++i) {
     sum += data[i];   
  }
  env->ReleaseIntArrayElements(src, data, 0);
  return sum;
}

It’s invalid to access the array(src) directly, so the native code has to call GetIntArrayElements to either pin the java array or make an copy from java array to native array. The reason that we have to call GetIntArrayElements is because another GC thread may move the data around during the native call. The Virtual Machine guarantees that the result of GetIntArrayElements points to a non-movable array of integers. The JVM will either "pin" down the array , or it will make a copy of the array into nonmovable memory. When the native code has finished using the array, it must call ReleaseIntArrayElements, which enables the JNI to copy back and free body if it is a copy of the original Java array, or "unpin" the Java array if it has been pinned in memory. Forgetting to do so will call memory leak.

If the array size is large and the code only needs to access part of the array, it can call Get/Set<type>ArrayRegion functions, which will just make a copy of the given region instead of the whole array

Another choice is to use GetPrimitiveArrayCritical, which is provided since Java 1.2. It will try to stop garbage collection on that array, and will provide direct access to the array pointer most of the time. The drawback of GetPrimitiveArrayCritical is that no blocking operation should be made between GetPrimitiveArrayCritical and ReleasePrimitiveArrayCritical, otherwise deadlock might happen. So code between the two calls should be treated as in critical section. Here is the code snippet:

static native int sumCritical(int[] src);
extern "C" JNIEXPORT jint JNICALL
Java_com_foo_test_JniPerfTest_sumCritical(
  JNIEnv* env, jclass, jintArray src) {
  const jint size = env->GetArrayLength(src);
  jint* data = (jint*) env->GetPrimitiveArrayCritical(src, 0);
  jint sum = 0;
  for (int i = 0; i < size; ++i) {
     sum += data[i];
  }
  env->ReleasePrimitiveArrayCritical(src, data, 0);
  return sum;
}

Compare that with an implementation of pure Java.

static int javaSum(int[] src) {
  int result = 0;
  for (int i = 0; i < src.length; ++i) {
    result += src[i];
  }
  return result;
}

Here the benchmark result running on a testing machine with 32bit vm(int array size is 1024). The time is the average time it takes to make a single sum function call.

Pure Java	JNI GetIntArrayElements	JNI GetPrimitiveArrayCritical
493 ns	1675 ns	703 ns

Although GetPrimitiveArrayCritical can significantly reduce the overhead of making a copy of array, the JNI implementation is still slower than the pure java implementation. Also, in some use cases, GetPrimitiveArrayCritical is not practical because of the potential blocking operation to the array or other data.

There are other drawbacks of JNI mentioned in various documents, like it exposes the raw access to pointer and make the program more vulnerable to invalid memory access, it’s hard to handle signal properly between java and native implementation, etc.

Summary

The usage of JNI should be limited because its performance overhead and other drawbacks like unsafe memory access.

If JNI is necessary, here is a few tips to make it more efficient:

• Avoid the number of JNI calls
• Limit the interaction between jvm and native code, limit the data passed around java code and native code(e.g use GetPrimitiveArrayCritical is possible).
• If the function’s performance is critical and the interaction between native code and java code is rare, consider making it intrinsic(let the JVM replace it with machine code directly at run-time, a typical example is System.arraycopy, a real example can be found here).

Real time profiling with PProf

Profiling Go Programs introduced how to profile a go program in a generic way. With the release of go 1, the profiling has become much easier: you can register various profile handlers to the running http server, and profile the program when it's running. The trick is very easy:

link the pprof http handler by adding the following line into your program(usually the main file):

      import _ "net/http/pprof"

Then you can easier grab head profile(understand how the memory is consumed) by using the commands below:

   go tool pprof http://localhost:6060/debug/pprof/heap

Or to look at a 30-second CPU profile:

      go tool pprof http://localhost:6060/debug/pprof/profile

Or to view all available profiles:

      go tool pprof http://localhost:6060/debug/pprof/

For more information on how the understand the pprof output, you can go to Profiling Go Programs or the homepage of google perf tools.

BTW: since there pprof handlers are so convenient, it is generally a good practice to start a debug http server with pprof enabled even if the program is not going to serve any http traffic.