Add LKM in Ariane using ariane-sdk

[0ena]

Hi,

I am trying to add a Linux Kernel Module inside the Linux image generated using the ariane-sdk of Ariane and I would like to know if the following steps are correct. After searching the topics opened about similar issues here in git, I found the following:

a) To add an LKM I need to either compile it directly in the Linux kernel or build a loadable kernel module and load it from the SD card on the rootfs. In the 2nd case, I need also to activate the support for loadable kernel modules #270.

b) In order to add the LKM, I will have to compile it with the riscv64-unknown-linux-gnu-gcc compiler and get the corresponding binary file (.elf). I will then need to copy this file in the ./rootfs directory of the ./ariane-sdk folder and build the new image (#25). In this way, I will be able to see the binary file inside Ariane's rootfs and run it. Am I correct on this? Is this considered the first case of compiling the LKM directly on the Linux kernel?

c) The LKM I am planning to add is pretty simple, basically it is this one. I want to use printk() and just print from inside the kernel. But I saw in #39, that printing from kernel is not possible. Is this really the case?

Thank you in advance for your help on this. Any advice will be greatly appreciated.

Kind regards,
Nassos

[Moschn]

a) Correct.
b) Kernel modules have to be compiled against the kernel tree, and they are not elf files but kernel modules .ko. The rest is correct.
c) Printing from the kernel is possible.

You probably will need to enable loadable kernel module support in the kernel first (with a KConfig flag before kernel compilation) before you are able to load your kernel module.

[0ena]

Thank you for your useful comments @Moschn! And for your help overall with our issues in this git, we really appreciate it.

I think I found a useful article to help me understand the KConfig flags concept: https://www.linuxjournal.com/content/kbuild-linux-kernel-build-system

I'll come back with any questions I might have. It is my first time going through this procedure.

[0ena]

Hi,

regarding enabling "loadable module support" I can see that there is an option in a Kconfig file in the following location ./ariane-sdk/buildroot/dl/linux/git/init/Kconfig that has the following lines:

1832 menuconfig MODULES
1833         bool "Enable loadable module support"
1834         option modules
1835         help
1836           Kernel modules are small pieces of compiled code which can
1837           be inserted in the running kernel, rather than being
1838           permanently built into the kernel.  You use the "modprobe"
1839           tool to add (and sometimes remove) them.  If you say Y here,
1840           many parts of the kernel can be built as modules (by
1841           answering M instead of Y where indicated): this is most
1842           useful for infrequently used options which are not required
1843           for booting.  For more information, see the man pages for
1844           modprobe, lsmod, modinfo, insmod and rmmod.
1845 
1846           If you say Y here, you will need to run "make
1847           modules_install" to put the modules under /lib/modules/
1848           where modprobe can find them (you may need to be root to do
1849           this).
1850 
1851           If unsure, say Y.
1852 
1853 if MODULES

I think this is the option that when enabled will load the modprobe programs (insmod, rmmod, etc.). Am I right on this? Does anyone know what change should I make in this file to enable it?

Also if I cd 2 folders back inside the ./ariane-sdk/buildroot/dl/linux/ folder, there is a git.readme file that has the following info:

IMPORTANT NOTE!

The git tree located in this directory is for the exclusive use
by Buildroot, which uses it as a local cache to reduce bandwidth
usage.

Buildroot *will* trash any changes in that tree whenever it needs
to use it. Buildroot may even remove it in case it detects the
repository may have been damaged or corrupted.

Do *not* work in that directory; your changes will eventually get
lost. Do *not* even use it as a remote, or as the source for new
worktrees; your commits will eventually get lost.

By that I suppose that even if I do any change inside the Kconfig file, it will be discarded by the Buildroot. How can I do the necessary change then?

Thank you in advance for your responses and time.

Kind regards,
Nassos

[0ena]

I think I am making some progress with this. I can see in the /PATHNAME/ariane-sdk/buildroot/dl/linux/git folder, in the Makefile, the following lines[1249:1326]:

# ---------------------------------------------------------------------------
# Modules

ifdef CONFIG_MODULES

# By default, build modules as well

all: modules

# Build modules
#
# A module can be listed more than once in obj-m resulting in
# duplicate lines in modules.order files.  Those are removed
# using awk while concatenating to the final file.

PHONY += modules
modules: $(vmlinux-dirs) $(if $(KBUILD_BUILTIN),vmlinux) modules.builtin
        $(Q)$(AWK) '!x[$$0]++' $(vmlinux-dirs:%=$(objtree)/%/modules.order) > $(objtree)/modules.order
        @$(kecho) '  Building modules, stage 2.';
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost

modules.builtin: $(vmlinux-dirs:%=%/modules.builtin)
        $(Q)$(AWK) '!x[$$0]++' $^ > $(objtree)/modules.builtin

%/modules.builtin: include/config/auto.conf include/config/tristate.conf
        $(Q)$(MAKE) $(modbuiltin)=$*


# Target to prepare building external modules
PHONY += modules_prepare
modules_prepare: prepare

# Target to install modules
PHONY += modules_install
modules_install: _modinst_ _modinst_post

PHONY += _modinst_
_modinst_:
        @rm -rf $(MODLIB)/kernel
        @rm -f $(MODLIB)/source
        @mkdir -p $(MODLIB)/kernel
        @ln -s $(abspath $(srctree)) $(MODLIB)/source
        @if [ ! $(objtree) -ef  $(MODLIB)/build ]; then \
                rm -f $(MODLIB)/build ; \
                ln -s $(CURDIR) $(MODLIB)/build ; \
        fi
        @cp -f $(objtree)/modules.order $(MODLIB)/
        @cp -f $(objtree)/modules.builtin $(MODLIB)/
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modinst

# This depmod is only for convenience to give the initial
# boot a modules.dep even before / is mounted read-write.  However the
# boot script depmod is the master version.
PHONY += _modinst_post
_modinst_post: _modinst_
        $(call cmd,depmod)

ifeq ($(CONFIG_MODULE_SIG), y)
PHONY += modules_sign
modules_sign:
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modsign
endif

else # CONFIG_MODULES

# Modules not configured
# ---------------------------------------------------------------------------

PHONY += modules modules_install
modules modules_install:
        @echo >&2
        @echo >&2 "The present kernel configuration has modules disabled."
        @echo >&2 "Type 'make config' and enable loadable module support."
        @echo >&2 "Then build a kernel with module support enabled."
        @echo >&2
        @exit 1

endif # CONFIG_MODULES

As stated at the end of the provided code above, if I execute in the same folder the command make config it starts providing several options to parametrize the kernel:

[user1@nassos2 git]$ make config
scripts/kconfig/conf  --oldaskconfig Kconfig
#
# using defaults found in /boot/config-3.10.0-1127.19.1.el7.x86_64
#
/boot/config-3.10.0-1127.19.1.el7.x86_64:697:warning: symbol value 'm' invalid for CPU_FREQ_STAT
/boot/config-3.10.0-1127.19.1.el7.x86_64:941:warning: symbol value 'm' invalid for NF_CT_PROTO_GRE
/boot/config-3.10.0-1127.19.1.el7.x86_64:969:warning: symbol value 'm' invalid for NF_NAT_REDIRECT
/boot/config-3.10.0-1127.19.1.el7.x86_64:972:warning: symbol value 'm' invalid for NF_TABLES_INET
/boot/config-3.10.0-1127.19.1.el7.x86_64:1139:warning: symbol value 'm' invalid for NF_TABLES_IPV4
/boot/config-3.10.0-1127.19.1.el7.x86_64:1143:warning: symbol value 'm' invalid for NF_TABLES_ARP
/boot/config-3.10.0-1127.19.1.el7.x86_64:1184:warning: symbol value 'm' invalid for NF_TABLES_IPV6
/boot/config-3.10.0-1127.19.1.el7.x86_64:1216:warning: symbol value 'm' invalid for NF_TABLES_BRIDGE
/boot/config-3.10.0-1127.19.1.el7.x86_64:1559:warning: symbol value 'm' invalid for NET_DEVLINK
/boot/config-3.10.0-1127.19.1.el7.x86_64:3664:warning: symbol value 'm' invalid for LIRC
*
* Linux/x86 5.1.0-rc7 Kernel Configuration
*
*
* Compiler: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
*
*
* General setup
*
Compile also drivers which will not load (COMPILE_TEST) [N/y/?] (NEW) N
Local version - append to kernel release (LOCALVERSION) [] 
Automatically append version information to the version string (LOCALVERSION_AUTO) [N/y/?] 
Build ID Salt (BUILD_SALT) [] (NEW) 
Kernel compression mode
> 1. Gzip (KERNEL_GZIP)
  2. Bzip2 (KERNEL_BZIP2)
  3. LZMA (KERNEL_LZMA)
  4. XZ (KERNEL_XZ)
  5. LZO (KERNEL_LZO)
  6. LZ4 (KERNEL_LZ4) (NEW)
choice[1-6?]: 
Default hostname (DEFAULT_HOSTNAME) [(none)] 
Support for paging of anonymous memory (swap) (SWAP) [Y/n/?] 
System V IPC (SYSVIPC) [Y/n/?] 
POSIX Message Queues (POSIX_MQUEUE) [Y/n/?] 
Enable process_vm_readv/writev syscalls (CROSS_MEMORY_ATTACH) [Y/n/?] 
uselib syscall (USELIB) [Y/n/?] (NEW) 
Auditing support (AUDIT) [Y/n/?] 
...
...
...
...

For the above, I need to respond with a Y/N to parametrize the kernel. I went over a lot of them and I suppose that at some point it will prompt me to enable/disable the support for loadable modules, but since I am no expert at this, it is risky to do it manually.

Then I found that in the ariane-sdk/buildroot/dl/linux/git/arch/riscv/configs folder, the variable CONFIG_MODULES is not set/defined. However this is an automatically generated file and it states that it should not be edited.
I suppose that if I set the CONFIG_MODULES variable, it will enable the loadable modules support.

Which file do I need to alter in order to set the CONFIG_MODULES variable?

Thank you for your help on this.

Kind regards,
Nassos

[0ena]

I think I solved it and I was able to load the modprobe tool inside the Linux kernel to enable the loadable module support.
There are configuration files inside in the /ariane-sdk/configs/ folder that are used to configure the compilation of the Linux kernel. Inside the busybox.config file, there are flags for the Linux Module Utilities that were not set. I changed them and now they look like the following:

#
# Linux Module Utilities
#
CONFIG_MODPROBE_SMALL=y
CONFIG_DEPMOD=y
CONFIG_INSMOD=y
CONFIG_LSMOD=y
CONFIG_FEATURE_LSMOD_PRETTY_2_6_OUTPUT=y
CONFIG_MODINFO=y
CONFIG_MODPROBE=y
CONFIG_FEATURE_MODPROBE_BLACKLIST=y
CONFIG_RMMOD=y

I must say that the file is a bit misleading because there is a comment on top that mentions that the file is autogenerated and not for editing, although it is used as is inside the Makefile.
Anyway, by making the above change and building my image as stated in the ariane-sdk, I managed to create an image that has the modprobe tools under the /sbin folder:

# ls /sbin/
acpid              ifdown             mke2fs             rtpr
adjtimex           ifstat             mkfs.ext2          rtstat
arp                ifup               mkfs.minix         run-init
blkid              init               mkfs.vfat          setconsole
blockdev           insmod             mkswap             ss
bridge             ip                 modinfo            start-stop-daemon
ctstat             ipaddr             modprobe           sulogin
depmod             iplink             mount.nfs          swapoff
devmem             ipneigh            mount.nfs4         swapon
fbsplash           iproute            nstat              switch_root
findfs             iprule             osd_login          sysctl
freeramdisk        iptunnel           pivot_root         syslogd
fsck               klogd              poweroff           tc
fsck.minix         lnstat             raidautorun        udhcpc
fstrim             loadkmap           reboot             uevent
genl               logread            rmmod              umount.nfs
getty              losetup            routef             umount.nfs4
hwclock            lsmod              routel             watchdog
ifcfg              makedevs           rtacct
ifconfig           mkdosfs            rtmon
# 

I would really appreciate it if someone could confirm that this is how we enable loadable module support, so I can close this issue.

Thank you in advance for your responses.

Kind regards,
Nassos

[Moschn]

Seems correct to me

[Juan-Gg]

(Solved below) I am trying to load a kernel module. I have enabled the Module Utilities flags on what is now cva6-sdk/configs/busybox64.config as done in this issue, and now lsmod, rmmod and modprobe are recognized as commands, and I am able to insmod a module.

But I cannot rmmod said module, or use modprobe -r.

Here is what I tried:

• modprobe gave a "can't change directory to '': No such file or directory" error, which I fixed by setting CONFIG_DEFAULT_MODULES_DIR="/lib/modules" in busybox64.config.

• rmmod gives me the following error:

# lsmod
hello_1 986 - - Live 0xffffffdf81307000 (O)
hello_2 986 - - Live 0xffffffdf81303000 (O)
# rmmod hello_1
rmmod: remove 'hello_1': Function not implemented

I believe this has something to do with the CONFIG_MODULE_UNLOAD parameter.

In ./buildroot/dl/linux/git/.config, CONFIG_MODULE_UNLOAD is set:

# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y

Yet in /buildroot/output/build/linux-v5.10.7/.config it is not

# CONFIG_MODULE_FORCE_LOAD is not set
# CONFIG_MODULE_UNLOAD is not set

I suppose the latter is the one that is actually implemented.

Here is my driver code and Makefile:

My driver code is the following from here

#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>

 static int __init hello_init(void)
 {
 	printk("----------->Hello, world\n");
 	return 0;
 }
 static void __exit hello_exit(void)
 {
 	printk("------------>Goodbye, cruel world\n");
 }
 module_init(hello_init);
 module_exit(hello_exit);

MODULE_LICENSE("GPL");

My makefile is as follows adapted from here:

XLEN 	:= 64

PWD := $(shell pwd)

ARCH := riscv
CROSS_COMPILE := $(PWD)/../../buildroot/output/host/bin/riscv$(XLEN)-buildroot-linux-gnu-
obj-m := hello-1.o
KDIR := ../../buildroot/output/build/linux-v5.10.7

export

default:
    $(MAKE) -C $(KDIR) M=$(PWD) modules

clean:
    $(MAKE) -C $(KDIR) M=$(PWD) clean

[Juan-Gg]

As it often happens, all it takes to solve something is to post an issue...

I added the following to cva6-sdk/configs/linux64_defconfig:

CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y

And after re-building the image, this is ./buildroot/output/build/linux-v5.10.7/.config:

...
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
# CONFIG_MODVERSIONS is not set
...

After building the module again (new kernel), it works (save for the dual-printing, which is a separate issue):

# insmod h2.ko
[  887.197103] hello_1: loading out-of-tree module taints kernel.
[  887.197103] hello_1: loading out-of-tree module taints kernel.
[  887.211946] ----------->Hello, world
[  887.211946] ----------->Hello, world
# rmmod hello_1
[  892.161913] ------------>Goodbye, cruel world
[  892.161913] ------------>Goodbye, cruel world
#