Simple Rule makes simple progress

 Test

Get the max from bazel: use a dashboard to track the build status

Build the dashboard server:

$ ./output/bazel build //src/tools/dash:all

Run it locally

$ bazel-bin/src/tools/dash/dash

Then you can stream your build result to this local server:

$ bazel build --use_dash --dash_url http://localhost:8080 lte/base:all

Or deploy it on appengine
$ bazel-bin/src/tools/dash/dash.deploy app-engine-app-id

Then use
http://app-engine-app-id.appspot.com to access it.

Use Namespace to gain special capabilities for testing(without docker)

Starting with kernel 2.2, Linux divides the privileges traditionally associated with superuser into distinct units, known as capabilities, which can be independently enabled and disabled.  Capabilities are a per-thread attribute.

For example, to be able to configure network setting, instead of becoming root user, if the process has CAP_NET_ADMIN capability, the relevant syscall will be permitted by kernel.

More information about capabilities can be found in Linux man page: man 7 CAPABILITIES.


Linux provides the following namespaces

       Namespace   Constant        Isolates
       IPC         CLONE_NEWIPC    System V IPC, POSIX message queues
       Network     CLONE_NEWNET    Network devices, stacks, ports, etc.
       Mount       CLONE_NEWNS     Mount points
       PID         CLONE_NEWPID    Process IDs
       User        CLONE_NEWUSER   User and group IDs
       UTS         CLONE_NEWUTS    Hostname and NIS domain name


User namespaces isolate security-related identifiers and attributes, in particular, user IDs and group IDs (see credentials(7)), the root directory, keys (see keyctl(2)), and capabilities (see capabilities(7)).

When a new user namespace is created(either using clone or unshare), it starts with *a complete/full set of capabilities*.

But that's good enough for us to run some test program inside. Inspired by the namespace-sandbox.c tool from bazel, I wrote a simpler version new-network-namespace.c that just creates an empty user namespace with network namespace and launch a given program inside the namespace. Inside that program, the user can perform ifconfig/iptables operation without being the real root, and without
worrying about break the real system by accident.

For example:

$ ./new-network-namespace /bin/bash
root@myhost:~# ifconfig -a
lo        Link encap:Local Loopback
          LOOPBACK  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Please give it a try, :)

Build DPDK with bazel

bazel is the open sourced version of Google build tool, and it has proved itself is a nice solution for monolithic server side development. DPDK is a nice solution for user mode packet processing. In our company, we use both to build our products and it greatly reduced the hurdle of rolling out any change because we put the whole DPDK source code into our source tree and the build process can easily produce final binaries at any point(i.e. we can put a fix to DPDK code and the build process will always produce up-to-date binary based on the change).

We pushed all our changes to github in "bazel" branch in case any one wants to try it. At the moment, due to the complexity of setting up bazel BUILD files to handle different compilation settings, we are not actively pursuing to integrate these changes back to upstream. But I'll be happy to answer any question and provide help in case anyone wants to do so.

Quickly turn gentoo daily snapshot into docker image

Download the snapshot from gentoo mirror:

Then
$ bunzip2 stage3-amd64-20141120.tar.bz2 -c | docker import - gentoo-amd64
7dbd254474e511597f160342bf8d828406f52467a62061b29ea0b3009b806b05

That's it!

Write zookeeper client log using google glog

I don't like the fact every other piece of log in my program is happily written by google glog but zookeeper is messing stderr by writing the log there. Besides, I like the format of glog better so I made a quick change to ask zookeeper c clien to write log using glog, here is a quick patch:

diff --git a/third-party/zookeeper/src/c/include/zookeeper_log.h b/third-party/zookeeper/src/c/include/zookeeper_log.h
index e5917cb..6519587 100644
--- a/third-party/zookeeper/src/c/include/zookeeper_log.h
+++ b/third-party/zookeeper/src/c/include/zookeeper_log.h
@@ -28,6 +28,24 @@ extern "C" {
 extern ZOOAPI ZooLogLevel logLevel;
 #define LOGSTREAM getLogStream()

+#define ZOOKEEPER_GLOG
+#ifdef ZOOKEEPER_GLOG
+ZOOAPI void zk_glog_message(int curLevel, int line, const char* funcName,
+                            const char* message);
+/* We can't include glog/log_severity.h because it's in C++ style, so
+ * we hard code the corresponding log level here */
+#define LOG_ERROR(x) if(logLevel>=ZOO_LOG_LEVEL_ERROR) \
+    zk_glog_message(2, __LINE__, __FILE__, format_log_message x)
+#define LOG_WARN(x) if(logLevel>=ZOO_LOG_LEVEL_WARN) \
+    zk_glog_message(1, __LINE__, __FILE__, format_log_message x)
+#define LOG_INFO(x) if(logLevel>=ZOO_LOG_LEVEL_INFO) \
+    zk_glog_message(0, __LINE__, __FILE__, format_log_message x)
+#define LOG_DEBUG(x) if(logLevel==ZOO_LOG_LEVEL_DEBUG) \
+    zk_glog_message(0, __LINE__, __FILE__, format_log_message x)
+#else
+
 #define LOG_ERROR(x) if(logLevel>=ZOO_LOG_LEVEL_ERROR) \
     log_message(ZOO_LOG_LEVEL_ERROR,__LINE__,__func__,format_log_message x)
 #define LOG_WARN(x) if(logLevel>=ZOO_LOG_LEVEL_WARN) \
@@ -36,6 +54,7 @@ extern ZOOAPI ZooLogLevel logLevel;
     log_message(ZOO_LOG_LEVEL_INFO,__LINE__,__func__,format_log_message x)
 #define LOG_DEBUG(x) if(logLevel==ZOO_LOG_LEVEL_DEBUG) \
     log_message(ZOO_LOG_LEVEL_DEBUG,__LINE__,__func__,format_log_message x)
+#endif

 ZOOAPI void log_message(ZooLogLevel curLevel, int line,const char* funcName,
     const char* message);
diff --git a/third-party/zookeeper/src/c/src/zk_glog.cc b/third-party/zookeeper/src/c/src/zk_glog.cc
new file mode 100644
index 0000000..ad2bdb3
--- /dev/null
+++ b/third-party/zookeeper/src/c/src/zk_glog.cc
@@ -0,0 +1,10 @@
+#include "third-party/google/glog/logging.h"
+#include "zookeeper_log.h"
+
+extern "C" {
+void zk_glog_message(int curLevel, int line, const char* filename,
+                     const char* message) {
+  LogMessage(filename, line, curLevel).stream() << message;
+}
+}

Run X11 application inside docker without VNC or SSH

I'm so annoyed that people would think of using ssh forwarding or VNC to run a X11 application inside docker. Docker is just a special chroot environment so there must be a more efficient way to setup the communication tunnel required by X11, and here it is:

docker run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY

/tmp/.X11-unix contains the unix domain socket file used by your running X11 application, but mounting it inside docker, the X11 app inside docker will  happily access it.