Provide in-kernel headers to make extending kernel easier
Introduce in-kernel headers which are made available as an archive
through proc (/proc/kheaders.tar.xz file). This archive makes it
possible to run eBPF and other tracing programs that need to extend the
kernel for tracing purposes without any dependency on the file system
having headers.
A github PR is sent for the corresponding BCC patch at:
https://github.com/iovisor/bcc/pull/2312
On Android and embedded systems, it is common to switch kernels but not
have kernel headers available on the file system. Further once a
different kernel is booted, any headers stored on the file system will
no longer be useful. This is an issue even well known to distros.
By storing the headers as a compressed archive within the kernel, we can
avoid these issues that have been a hindrance for a long time.
The best way to use this feature is by building it in. Several users
have a need for this, when they switch debug kernels, they do not want to
update the filesystem or worry about it where to store the headers on
it. However, the feature is also buildable as a module in case the user
desires it not being part of the kernel image. This makes it possible to
load and unload the headers from memory on demand. A tracing program can
load the module, do its operations, and then unload the module to save
kernel memory. The total memory needed is 3.3MB.
By having the archive available at a fixed location independent of
filesystem dependencies and conventions, all debugging tools can
directly refer to the fixed location for the archive, without concerning
with where the headers on a typical filesystem which significantly
simplifies tooling that needs kernel headers.
The code to read the headers is based on /proc/config.gz code and uses
the same technique to embed the headers.
Other approaches were discussed such as having an in-memory mountable
filesystem, but that has drawbacks such as requiring an in-kernel xz
decompressor which we don't have today, and requiring usage of 42 MB of
kernel memory to host the decompressed headers at anytime. Also this
approach is simpler than such approaches.
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-04-27 03:04:29 +08:00
|
|
|
#!/bin/bash
|
|
|
|
|
# SPDX-License-Identifier: GPL-2.0
|
|
|
|
|
|
|
|
|
|
# This script generates an archive consisting of kernel headers
|
2019-05-16 05:35:51 +08:00
|
|
|
# for CONFIG_IKHEADERS.
|
Provide in-kernel headers to make extending kernel easier
Introduce in-kernel headers which are made available as an archive
through proc (/proc/kheaders.tar.xz file). This archive makes it
possible to run eBPF and other tracing programs that need to extend the
kernel for tracing purposes without any dependency on the file system
having headers.
A github PR is sent for the corresponding BCC patch at:
https://github.com/iovisor/bcc/pull/2312
On Android and embedded systems, it is common to switch kernels but not
have kernel headers available on the file system. Further once a
different kernel is booted, any headers stored on the file system will
no longer be useful. This is an issue even well known to distros.
By storing the headers as a compressed archive within the kernel, we can
avoid these issues that have been a hindrance for a long time.
The best way to use this feature is by building it in. Several users
have a need for this, when they switch debug kernels, they do not want to
update the filesystem or worry about it where to store the headers on
it. However, the feature is also buildable as a module in case the user
desires it not being part of the kernel image. This makes it possible to
load and unload the headers from memory on demand. A tracing program can
load the module, do its operations, and then unload the module to save
kernel memory. The total memory needed is 3.3MB.
By having the archive available at a fixed location independent of
filesystem dependencies and conventions, all debugging tools can
directly refer to the fixed location for the archive, without concerning
with where the headers on a typical filesystem which significantly
simplifies tooling that needs kernel headers.
The code to read the headers is based on /proc/config.gz code and uses
the same technique to embed the headers.
Other approaches were discussed such as having an in-memory mountable
filesystem, but that has drawbacks such as requiring an in-kernel xz
decompressor which we don't have today, and requiring usage of 42 MB of
kernel memory to host the decompressed headers at anytime. Also this
approach is simpler than such approaches.
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-04-27 03:04:29 +08:00
|
|
|
set -e
|
|
|
|
|
spath="$(dirname "$(readlink -f "$0")")"
|
|
|
|
|
kroot="$spath/.."
|
|
|
|
|
outdir="$(pwd)"
|
|
|
|
|
tarfile=$1
|
|
|
|
|
cpio_dir=$outdir/$tarfile.tmp
|
|
|
|
|
|
|
|
|
|
# Script filename relative to the kernel source root
|
|
|
|
|
# We add it to the archive because it is small and any changes
|
|
|
|
|
# to this script will also cause a rebuild of the archive.
|
|
|
|
|
sfile="$(realpath --relative-to $kroot "$(readlink -f "$0")")"
|
|
|
|
|
|
|
|
|
|
src_file_list="
|
|
|
|
|
include/
|
|
|
|
|
arch/$SRCARCH/include/
|
|
|
|
|
$sfile
|
|
|
|
|
"
|
|
|
|
|
|
|
|
|
|
obj_file_list="
|
|
|
|
|
include/
|
|
|
|
|
arch/$SRCARCH/include/
|
|
|
|
|
"
|
|
|
|
|
|
|
|
|
|
# Support incremental builds by skipping archive generation
|
|
|
|
|
# if timestamps of files being archived are not changed.
|
|
|
|
|
|
|
|
|
|
# This block is useful for debugging the incremental builds.
|
|
|
|
|
# Uncomment it for debugging.
|
2019-05-16 05:35:52 +08:00
|
|
|
# if [ ! -f /tmp/iter ]; then iter=1; echo 1 > /tmp/iter;
|
|
|
|
|
# else iter=$(($(cat /tmp/iter) + 1)); echo $iter > /tmp/iter; fi
|
2019-07-01 08:58:43 +08:00
|
|
|
# find $src_file_list -type f | xargs ls -l > /tmp/src-ls-$iter
|
|
|
|
|
# find $obj_file_list -type f | xargs ls -l > /tmp/obj-ls-$iter
|
Provide in-kernel headers to make extending kernel easier
Introduce in-kernel headers which are made available as an archive
through proc (/proc/kheaders.tar.xz file). This archive makes it
possible to run eBPF and other tracing programs that need to extend the
kernel for tracing purposes without any dependency on the file system
having headers.
A github PR is sent for the corresponding BCC patch at:
https://github.com/iovisor/bcc/pull/2312
On Android and embedded systems, it is common to switch kernels but not
have kernel headers available on the file system. Further once a
different kernel is booted, any headers stored on the file system will
no longer be useful. This is an issue even well known to distros.
By storing the headers as a compressed archive within the kernel, we can
avoid these issues that have been a hindrance for a long time.
The best way to use this feature is by building it in. Several users
have a need for this, when they switch debug kernels, they do not want to
update the filesystem or worry about it where to store the headers on
it. However, the feature is also buildable as a module in case the user
desires it not being part of the kernel image. This makes it possible to
load and unload the headers from memory on demand. A tracing program can
load the module, do its operations, and then unload the module to save
kernel memory. The total memory needed is 3.3MB.
By having the archive available at a fixed location independent of
filesystem dependencies and conventions, all debugging tools can
directly refer to the fixed location for the archive, without concerning
with where the headers on a typical filesystem which significantly
simplifies tooling that needs kernel headers.
The code to read the headers is based on /proc/config.gz code and uses
the same technique to embed the headers.
Other approaches were discussed such as having an in-memory mountable
filesystem, but that has drawbacks such as requiring an in-kernel xz
decompressor which we don't have today, and requiring usage of 42 MB of
kernel memory to host the decompressed headers at anytime. Also this
approach is simpler than such approaches.
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-04-27 03:04:29 +08:00
|
|
|
|
|
|
|
|
# include/generated/compile.h is ignored because it is touched even when none
|
|
|
|
|
# of the source files changed. This causes pointless regeneration, so let us
|
|
|
|
|
# ignore them for md5 calculation.
|
|
|
|
|
pushd $kroot > /dev/null
|
|
|
|
|
src_files_md5="$(find $src_file_list -type f |
|
|
|
|
|
grep -v "include/generated/compile.h" |
|
2019-05-16 05:35:52 +08:00
|
|
|
grep -v "include/generated/autoconf.h" |
|
|
|
|
|
grep -v "include/config/auto.conf" |
|
|
|
|
|
grep -v "include/config/auto.conf.cmd" |
|
|
|
|
|
grep -v "include/config/tristate.conf" |
|
2019-07-01 08:58:43 +08:00
|
|
|
xargs ls -l | md5sum | cut -d ' ' -f1)"
|
Provide in-kernel headers to make extending kernel easier
Introduce in-kernel headers which are made available as an archive
through proc (/proc/kheaders.tar.xz file). This archive makes it
possible to run eBPF and other tracing programs that need to extend the
kernel for tracing purposes without any dependency on the file system
having headers.
A github PR is sent for the corresponding BCC patch at:
https://github.com/iovisor/bcc/pull/2312
On Android and embedded systems, it is common to switch kernels but not
have kernel headers available on the file system. Further once a
different kernel is booted, any headers stored on the file system will
no longer be useful. This is an issue even well known to distros.
By storing the headers as a compressed archive within the kernel, we can
avoid these issues that have been a hindrance for a long time.
The best way to use this feature is by building it in. Several users
have a need for this, when they switch debug kernels, they do not want to
update the filesystem or worry about it where to store the headers on
it. However, the feature is also buildable as a module in case the user
desires it not being part of the kernel image. This makes it possible to
load and unload the headers from memory on demand. A tracing program can
load the module, do its operations, and then unload the module to save
kernel memory. The total memory needed is 3.3MB.
By having the archive available at a fixed location independent of
filesystem dependencies and conventions, all debugging tools can
directly refer to the fixed location for the archive, without concerning
with where the headers on a typical filesystem which significantly
simplifies tooling that needs kernel headers.
The code to read the headers is based on /proc/config.gz code and uses
the same technique to embed the headers.
Other approaches were discussed such as having an in-memory mountable
filesystem, but that has drawbacks such as requiring an in-kernel xz
decompressor which we don't have today, and requiring usage of 42 MB of
kernel memory to host the decompressed headers at anytime. Also this
approach is simpler than such approaches.
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-04-27 03:04:29 +08:00
|
|
|
popd > /dev/null
|
|
|
|
|
obj_files_md5="$(find $obj_file_list -type f |
|
|
|
|
|
grep -v "include/generated/compile.h" |
|
2019-05-16 05:35:52 +08:00
|
|
|
grep -v "include/generated/autoconf.h" |
|
|
|
|
|
grep -v "include/config/auto.conf" |
|
|
|
|
|
grep -v "include/config/auto.conf.cmd" |
|
|
|
|
|
grep -v "include/config/tristate.conf" |
|
2019-07-01 08:58:43 +08:00
|
|
|
xargs ls -l | md5sum | cut -d ' ' -f1)"
|
Provide in-kernel headers to make extending kernel easier
Introduce in-kernel headers which are made available as an archive
through proc (/proc/kheaders.tar.xz file). This archive makes it
possible to run eBPF and other tracing programs that need to extend the
kernel for tracing purposes without any dependency on the file system
having headers.
A github PR is sent for the corresponding BCC patch at:
https://github.com/iovisor/bcc/pull/2312
On Android and embedded systems, it is common to switch kernels but not
have kernel headers available on the file system. Further once a
different kernel is booted, any headers stored on the file system will
no longer be useful. This is an issue even well known to distros.
By storing the headers as a compressed archive within the kernel, we can
avoid these issues that have been a hindrance for a long time.
The best way to use this feature is by building it in. Several users
have a need for this, when they switch debug kernels, they do not want to
update the filesystem or worry about it where to store the headers on
it. However, the feature is also buildable as a module in case the user
desires it not being part of the kernel image. This makes it possible to
load and unload the headers from memory on demand. A tracing program can
load the module, do its operations, and then unload the module to save
kernel memory. The total memory needed is 3.3MB.
By having the archive available at a fixed location independent of
filesystem dependencies and conventions, all debugging tools can
directly refer to the fixed location for the archive, without concerning
with where the headers on a typical filesystem which significantly
simplifies tooling that needs kernel headers.
The code to read the headers is based on /proc/config.gz code and uses
the same technique to embed the headers.
Other approaches were discussed such as having an in-memory mountable
filesystem, but that has drawbacks such as requiring an in-kernel xz
decompressor which we don't have today, and requiring usage of 42 MB of
kernel memory to host the decompressed headers at anytime. Also this
approach is simpler than such approaches.
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-04-27 03:04:29 +08:00
|
|
|
|
|
|
|
|
if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
|
|
|
|
|
if [ -f kernel/kheaders.md5 ] &&
|
|
|
|
|
[ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
|
|
|
|
|
[ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&
|
|
|
|
|
[ "$(cat kernel/kheaders.md5|tail -1)" == "$tarfile_md5" ]; then
|
|
|
|
|
exit
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
if [ "${quiet}" != "silent_" ]; then
|
|
|
|
|
echo " GEN $tarfile"
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
rm -rf $cpio_dir
|
|
|
|
|
mkdir $cpio_dir
|
|
|
|
|
|
|
|
|
|
pushd $kroot > /dev/null
|
|
|
|
|
for f in $src_file_list;
|
|
|
|
|
do find "$f" ! -name "*.cmd" ! -name ".*";
|
|
|
|
|
done | cpio --quiet -pd $cpio_dir
|
|
|
|
|
popd > /dev/null
|
|
|
|
|
|
|
|
|
|
# The second CPIO can complain if files already exist which can
|
|
|
|
|
# happen with out of tree builds. Just silence CPIO for now.
|
|
|
|
|
for f in $obj_file_list;
|
|
|
|
|
do find "$f" ! -name "*.cmd" ! -name ".*";
|
|
|
|
|
done | cpio --quiet -pd $cpio_dir >/dev/null 2>&1
|
|
|
|
|
|
|
|
|
|
# Remove comments except SDPX lines
|
|
|
|
|
find $cpio_dir -type f -print0 |
|
|
|
|
|
xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
|
|
|
|
|
|
|
|
|
|
tar -Jcf $tarfile -C $cpio_dir/ . > /dev/null
|
|
|
|
|
|
2019-05-16 05:35:52 +08:00
|
|
|
echo "$src_files_md5" > kernel/kheaders.md5
|
Provide in-kernel headers to make extending kernel easier
Introduce in-kernel headers which are made available as an archive
through proc (/proc/kheaders.tar.xz file). This archive makes it
possible to run eBPF and other tracing programs that need to extend the
kernel for tracing purposes without any dependency on the file system
having headers.
A github PR is sent for the corresponding BCC patch at:
https://github.com/iovisor/bcc/pull/2312
On Android and embedded systems, it is common to switch kernels but not
have kernel headers available on the file system. Further once a
different kernel is booted, any headers stored on the file system will
no longer be useful. This is an issue even well known to distros.
By storing the headers as a compressed archive within the kernel, we can
avoid these issues that have been a hindrance for a long time.
The best way to use this feature is by building it in. Several users
have a need for this, when they switch debug kernels, they do not want to
update the filesystem or worry about it where to store the headers on
it. However, the feature is also buildable as a module in case the user
desires it not being part of the kernel image. This makes it possible to
load and unload the headers from memory on demand. A tracing program can
load the module, do its operations, and then unload the module to save
kernel memory. The total memory needed is 3.3MB.
By having the archive available at a fixed location independent of
filesystem dependencies and conventions, all debugging tools can
directly refer to the fixed location for the archive, without concerning
with where the headers on a typical filesystem which significantly
simplifies tooling that needs kernel headers.
The code to read the headers is based on /proc/config.gz code and uses
the same technique to embed the headers.
Other approaches were discussed such as having an in-memory mountable
filesystem, but that has drawbacks such as requiring an in-kernel xz
decompressor which we don't have today, and requiring usage of 42 MB of
kernel memory to host the decompressed headers at anytime. Also this
approach is simpler than such approaches.
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-04-27 03:04:29 +08:00
|
|
|
echo "$obj_files_md5" >> kernel/kheaders.md5
|
|
|
|
|
echo "$(md5sum $tarfile | cut -d ' ' -f1)" >> kernel/kheaders.md5
|
|
|
|
|
|
|
|
|
|
rm -rf $cpio_dir
|