October 28, 2016 - Mauro Carvalho Chehab
Improving Linux Kernel Development Process Documentation
This article will cover how the Linux kernel community handled the conversion of documentation related to the kernel development process; it’s part of a series on improvements being made to Linux kernel documentation.
It’s not an easy task to properly describe the Linux development process. The kernel community moves at a very fast pace and produces about 6 versions per year. Thousands of people, distributed worldwide, contribute to this collective work; the development process is a live being that constantly adjusts to what best fits the people involved in the process. Additionally, since kernel development is managed per subsystems, each maintainer has their own criteria for what works best for the subsystem they take care of. To address this, the documentation provides a common ground for understanding the best practices all kernel developers should follow.
The Documentation/Development-Process Book
There are several files inside the kernel that describes the development process. In 2008, the Linux Foundation sponsored work from Jonathan Corbet to create a set of files that group the best development practices together into a development-process book. This book received additional updates in 2011 to describe the changes that happened since the original version, and reflect the practices followed for kernel 2.6.38. The kernel development practices for kernels 4.x are aproximately the same as they were during the late 2.6.x kernels; this document is not too far from the current practices.
The Essential Documentation for Kernel Developers
All kernel developers use a set of well-known documents that were written before the development-process files. These contain, among other things, the procedures to submit patches and drivers, the logic used to apply patches, how to solve conflicts, and the description of the kernel coding style. All kernel developers should be familiar with these documents, yet they’re in the middle of several other documents that include driver descriptions.
In order to get some sanity out of the chaos, I selected a group of documents that, in my opinion, contain the essence of the knowledge that all kernel developer should know, and added them into the development-process book:
Documentation/HOWTOis a sort of index for the other files in this set, it provides an introduction to Linux kernel development,
Documentation/adding-syscalls.txtdescribes the steps involved in adding a new system call to the Linux kernel,
Documentation/applying-patches.txtdescribes how to apply a patch to the Linux kernel using the patch tool, also describes the patch files stored at ftp.kernel.org
Documentation/Changesdespite the name, describes the minimal requirement to build and to run the Linux kernel,
Documentation/CodeOfConflictdescribes the process to be used if someone feels personally abused, threatened, or otherwise uncomfortable with the development process,
Documentation/CodingStyledescribes the preferred coding style for the C files inside the Linux kernel tree,
Documentation/email-clients.txtprovides recommendations for developers to use
git send-emailto send e-mails with patches whenever possible. When not possible, provides information about how to setup e-mail clients to avoid mangling patches,
Documentation/kernel-docs.txtcontains a list of external references and books related to the kernel development,
Documentation/magic-number.txtprovides the set of magic numbers used inside the Linux kernel to help to protect kernel data structures,
Documentation/ManagementStyleprovides a general guideline about the prefered management style for the Linux kernel,
Documentation/stable_api_nonsense.txtexplains why the kernel internal API/ABI doesn’t have a stable kernel interface, and the advantages of such an approach,
Documentation/stable_kernel_rules.txtexplains the process related to the kernel stable releases and the related development process,
Documentation/SubmitChecklistcontains a list of actions that should be followed before submitting a new patch,
Documentation/SubmittingDriversdescribes the process of submitting a new driver to the kernel,
Documentation/SubmittingPatchesdescribes the process of submitting a new driver to the kernel, and
Documentation/volatile-considered-harmful.txtexplains the proper procedure to work with data that could be modified by some other CPU or task (“volatile”).
There are other files that might be considered as part of the development process in the future.
Converting Kernel Development Files to Sphinx
As part of the effort to convert Linux documentation to Sphinx we converted the files listed above. Most of the work will be merged for kernel 4.9, but some conversions won’t be available until kernel 4.10.
Converting the Documentation/development-process files to a Sphinx book was trivial. First, Paragraph markups were added to the chapters and sections. Then, special characters that Sphinx uses were converted; specifically, emphasis required the correct markups. Finally, cross-references to other documents that were already converted to Sphinx were added.
The conversion of the other files was somewhat tricky. They’re all part of the development process, so ideally they should be located in the same directory: Documentation/development-process. However, moving several files is something that should be carefully considered, as some of them (like: CodingStyle, SubmittingDrivers, SubmittingPatches) are well known within the developer community. Moving them to any other directory would break a wide range of links that point to them, not only at the kernel tree itself, but also hyper-links all over the web. Worse than that, it will also break people’s mental references to those import documents. So, the approach should avoid physically moving them as much as possible, this results in a few choices:
- copy the files to the new directory, replacing the original contents with a link to the new place,
- create the files in the new location and use the Sphinx include directive, or
- use soft links at the new location.
Option 1 is the best long-term solution because the number of “orphan” files under Documentation/ will be reduced, making it easier to identify the files that haven’t been converted to Sphinx yet. Option 2 would require creating one file for every file that’s merged. Alternatively, the Sphinx include statements could be in a single RST file. However, in this case a large set of documentation files would be in the same HTML file since Sphinx generates only one output HTML file per RST source. Option 3 is easy to do and doesn’t prevent the adoption of another option in the future, so we opted to follow this direction. However, this has made it more challenging to figure out which files under
Documentation/ haven’t been converted to Sphinx nor merged into a book, so we may need to review this decision in the future.
About Mauro Carvalho Chehab
Mauro is the upstream maintainer of the Linux kernel media and EDAC subsystems, and also a major contributor for the Reliability Availability and Serviceability (RAS) subsystems. Mauro also maintains Tizen on Yocto packages upstream. He works for the Samsung Open Source Group since 2013. Has worked during 5 years at the Red Hat RHEL Kernel team, having broad experience in telecommunications due to his work at some of the Brazilian largest wired and wireless carriers.