Tuesday, December 11, 2012

Contiki Regression Tests: 9 Hardware Platforms, 4 Processor Achitectures, 1021 Network Nodes

tl;dr: Contiki gets regression test framework from Thingsquare with travis integration that lets us test every new commit on 9 hardware platforms, 4 processor architectures, and 1021 emulated network nodes.

Despite its size, Contiki a complex system with multiple layers of interrupts, processes, protothreads, serial port input and output functions, radio device drivers, power-saving duty cycling mechanisms, medium access control protocols, multiple network stacks, fragmentation techniques, self-healing network routing protocols, best-effort and reliable communication abstractions, and Internet application protocols. These run on a wide range of different microprocessor architectures, hardware devices, and is compiled with a variety of C compilers.

Typical Contiki systems also have extreme memory constraints and form large, unreliable wireless networks. How can we ensure that Contiki, with all these challenges, does what it is supposed to do?

Over the years, open source projects have tried different ways to ensure that the code always is stable across multiple platforms. A common approach has been to ask people to test the code on their own favorite hardware in good time before a release. This was the approach that Contiki took a few years ago. But the problem was that it is really hard to get good test coverage, particularly for systems that are inherently networked. Most testers won't have access to large numbers of nodes and even if they have, tests are difficult to set up because of the size of networks that are needed for testing. Also, since people are more motivated to run tests near a release, there may potentially be large numbers of bugs that are found right before the release. It would be great to be able to find those bugs much earlier.

Many projects do nightly builds to ensure that the source code is kept sane. This is something we have done for a long time in Contiki: the code has been compiled with 5 different C compilers for 12 platforms. But this is not enough to catch problems with code correctness, as the functionality of the system is not tested. Testing the functionality is much more difficult, since it requires us to actually run the code.

Fortunately, Contiki provides a way to run automated tests in large networks with a fine-grained level of detail: Cooja, the Contiki network simulator. But taking this to a full regression test framework took a bit of work.

First, to make scripted simulation setups easier, Cooja author Fredrik Österlind wrote a test script framework for Cooja. Second, Github contributors Rémy Léone and Ilya Dmitrichenko developed a travis plugin for Contiki. And now Contiki gets a new regression test framework from Thingsquare Mist.

Cooja, the Contiki network simulator

Cooja simulate networks of nodes, where each node can be a highly detailed emulator of a specific hardware device. Thanks to mspsim and avrora, Cooja can emulate the TI MSP430 CPU, with a recent addition of the MSP430x architecture, and the Atmel AVR. Nodes can also run non-emulated Contiki code compiled for the native platform, typically the x86.

With Cooja, the regression test framework, and travis, can now run regression tests on every single commit done to the repository. We spot problems early in the development process; even before the code gets into the repository. Although regression tests do not provide a full certainty that the code works in all situations, it does help us spot showstoppers. This lets us all sleep well at night, knowing that any new code in the system is not only reviewed by the Contiki developers and automatically built, but that the code is automatically run in a range of different network setups, with a number of different network protocols, and on multiple platforms and processor architectures.

The regression test framework resides inside directory in the Contiki root, regression-tests/. To run the complete set of regression tests, simply run make in this directory and all tests will run. Each test prints out either OK or FAIL ಠ_ಠ to indicate success or failure of each test.

The regression-tests/ directory contains a set of subdirectories:
01-compile/
02-hello-world/
03-base/
04-rime/
05-netperf/
06-shell/
07-elfloader/
08-collect/
09-ipv4/
10-ipv6/
11-rpl/
12-ipv6-apps/
The directory names start with two digits to create an explicit ordering and allow doing iterative regression testing. For example, if working on the RPL code, go into the 11-rpl/ directory and run make to check that the regression tests for RPL work, without having to run all the other tests that aren't relevant to the RPL code.

Each test directory contains a number of tests. Simulation test directories contain Cooja csc files, who's names also start with two digits. For example, the 02-hello-world/ directory contains:
Makefile
01-multithreading.csc
02-sky-coffee.csc
x03-crosslevel.csc
04-sky-checkpointing.csc
The files that start with two digits are run, in order, when running make in the directory. The file named x03-crosslevel.csc has been disabled, by putting an x in the beginning of the filename.

Some of the tests include hundreds of networked nodes, whereas others contain only a single node. For example, the multithreading and hello-world tests contain only one node each. Some of the IPv6/RPL and Rime tests contain hundreds of nodes, to test the scalability of the network protocols by checking that the behave correctly even if the number of neighbors and routes exhaust the neighbor and routing tables used by the network protocols.

A few statistics for the new regression test setup:
  • Number of automated tests: 42
  • Number of build platforms: 11
  • Number of compiled examples: 38
  • Number of C compilers used: 4
  • Number of processor configurations tested: 9
  • Number of processor architectures tested: 4 (AVR, MSP430, MSP430x, x86)
  • Number of network stacks tested: 3 (IPv4, IPv6, Rime)
  • Number of network protocols tested: 16
  • Total number of network nodes used in the tests: 1021
  • Average number of per-node neighbors: 252.7

Monday, October 29, 2012

Contiki goes Github

To make it easy for everyone to contribute their cool new features, improvements, and bugfixes, the Contiki project (finally) moves to Github! Read all about it in this email to the contiki-developers mailing list.

Fork the Contiki code at the Contiki github repository:


Want to contribute your coolest code to Contiki? Check out the instructions here:

Sunday, August 5, 2012

Getting a Contiki Q&A site started - your help is needed

Wouldn't it be nice to having a real Q&A site like stackoverflow.com for Contiki? The mailing list is great and all, but having something community moderated has a lot of advantages.

We have set up an Area51 site on stackexchange.com to get this started. Once it has enough followers, sample questions, and votes, then the site becomes live and experts can answer questions.

How can you help? Read all about how to help this get off the ground here:


In short, you need to join the site, follow it, ask 5 questions and vote up 5 questions. Pretty simple! The wiki is also tracking the posted questions so that if you think of more than 5 questions, you can write them down. And if you can't think of 5 questions, check the wiki. Later, after the "definition" stage, we will need answers. So in the meantime, you can post them to the wiki as well.

We hope this would work well as a great Q&A forum for Contiki. Let's make it happen!

New Contiki Wiki

We have set up a new Contiki community wiki here:


Unlike the previous wikis that Contiki has had, we now have an explicit Creative Commons license for all contributions. If you have added something to the old wiki, please move it to the new one to make sure it is covered by the new license.

Tuesday, July 17, 2012

Contiki 2.6 Released

The Contiki team is happy to announce the release of Contiki 2.6. This release brings several improvements to core Contiki networking components, such as the IPv6 RPL routing protocol and Erbuim, a new CoAP implementation, as well as a set of new platforms (cc2530, exp5438, wismote), and several new Cooja simulation platforms (exp5438, wismote, z1). The release also adds new Contiki applications, such as the Antelope low-power database manager and a new HTTP server for web services. Sleepy router support has also been extended to new platforms (STM32w).

Download the Contiki 2.6 from the Contiki web site:


Contiki is the open source operating system for the Internet of Things. Contiki allows tiny, battery-operated low-power systems communicate with the Internet. Contiki is used in a wide variety of systems such as city sound monitoring, street lights, networked electrical power meters, industrial monitoring, radiation monitoring, construction site monitoring, alarm systems, and remote house monitoring.

Contiki 2.6 changelog

  • Antelope. A lightweight SQL-like database manager for Contiki (apps/antelope and examples/antelope).
  • Erbium. A new CoAP implementation for Contiki (apps/erbium and examples/erbium).
  • JSON library. A lightweight JSON parsing and construction library (apps/json) and a web services HTTP server (apps/httpd-ws).
  • ContikiMAC. Improvements to allow busts of packets to be sent rapidly.
  • RPL. Added support for extension headers.
  • Native border router. The RPL border router can now optionally run as a native process under Linux and communicate with a low-power radio over a serial port.
  • Cooja. Many improvements to the user interface. Simulation support for the MSP430x architecture and the exp5438, wismote, and z1 platforms.
  • stm32w. Added support for ContikiMAC.
  • New ports: cc2530, exp5438, wismote.
  • Instant Contiki. Updated to Ubuntu 12.04. Eclipse installed as default editor.

Wednesday, July 11, 2012

A New Website for Contiki

Finally, a new website for Contiki! Redesigned, redone, restructured - check it out:

Tuesday, July 10, 2012

A New Beginning for Contiki

Change is about to come to Contiki. A new development model. Improved release cycles. Better communication. Stay tuned.

Read more in this message to contiki-developers.

Tuesday, February 14, 2012

How do you use Contiki?

A lot of people use Contiki for a variety of cool projects and products, but since Contiki is open source, it is difficult to know about them. I (Adam) regularly receive emails from people using Contiki in a variety of fascinating and inspiring projects, and often hear from people I meet about their cool Contiki-projects, but it would be great to get a more structured view of how Contiki is being used.

Are you using Contiki or thinking about using it? If so, it would be great if you could take a few moments of your time to fill out this poll:

How do you use Contiki?

Individual answers will be treated confidentially, and the aggregate results will be posted in early March. Or, if you prefer to not fill out the poll, but still would like to let me know how you use Contiki, feel free to drop me a direct email at adam@dunkels.com! I love to hear about all Contiki-related projects, be it commercial products, hobby projects, academic project, or something completely different!

The poll closes on February 29th 2012, so be quick to answer.

Wednesday, January 18, 2012

New Contiki Papers

A number of Contiki-related papers have been published recently: