Bug#1033301: linux: arm64 kernel size increased from 31 to 39 MB, causing u-boot-rpi to fail

[Salvatore Bonaccorso]

Hi Aurelien,

Thanks for tracking this down. I would like to loop in Masahiro and
upstream to see if something can/should be done on upstream side.

Masahiro, in short, upstream change 994b7ac1697b ("arm64: remove
special treatment for the link order of head.o") (which got backported
as well to 6.1.14) caused the vmlinuz size to icrease significantly,
causing boot failures on Raspberry Pi 3 Model B Plus with u-boot
parameters previously working. Full quoting the Debian report below

Source: linux
Version: 6.1.15-1
Severity: important
Tags: upstream
Control: affects -1 + u-boot-rpi
Hi,
Following the upgrade of the kernel from 6.1.12-1 to 6.1.15-1 on a
Raspberry Pi 3 Model B Plus, u-boot (from the u-boot-rpi package) failed
| 40175552 bytes read in 1695 ms (23 MiB/s)
| 43794863 bytes read in 1817 ms (23 MiB/s)
| Moving Image from 0x80000 to 0x200000, end=2990000
| ERROR: RD image overlaps OS image (OS=0x200000..0x2990000)
I tracked the issue to a significant increase of the kernel size between
| 31492 /boot/vmlinuz-6.1.0-5-arm64
| 39236 /boot/vmlinuz-6.1.0-6-arm64
This is more than the 36MB that is allowed by u-boot with the default
load addresses. A workaround is to shift the load addresses at the
u-boot level as in the attached patch.
I have tracked issue on the upstream kernel side to the following commit
| commit 3e3e4d234d46e48480a7c7c35399fa811182e8ef
| Date: Thu Oct 13 08:35:00 2022 +0900
|
| arm64: remove special treatment for the link order of head.o
|
| commit 994b7ac1697b4581b7726d2ac64321e3c840229b upstream.
|
| In the previous discussion (see the Link tag), Ard pointed out that
| arm/arm64/kernel/head.o does not need any special treatment - the only
| piece that must appear right at the start of the binary image is the
| image header which is emitted into .head.text.
|
| The linker script does the right thing to do. The build system does
| not need to manipulate the link order of head.o.
|
The problem is still reproducible on Linus' master.
I am reporting the bug to the linux package as I believed there is no
real reason for such an increase in the kernel size. In case I missed
something and this is actually wanted, the bug can be reassigned to the
u-boot package.
Regards
Aurelien
--- u-boot-2023.01+dfsg.orig/include/configs/rpi.h
+++ u-boot-2023.01+dfsg/include/configs/rpi.h
@@ -95,32 +95,32 @@
* text_offset bytes (specified in the header of the Image) into a 2MB
* boundary. The 'booti' command relocates the image if necessary. Linux uses
* a default text_offset of 0x80000. In summary, loading at 0x80000
- * satisfies all these constraints and reserving memory up to 0x02400000
- * permits fairly large (roughly 36M) kernels.
+ * satisfies all these constraints and reserving memory up to 0x02a00000
+ * permits fairly large (roughly 42M) kernels.
*
* scriptaddr and pxefile_addr_r can be pretty much anywhere that doesn't
* conflict with something else. Reserving 1M for each of them at
- * 0x02400000-0x02500000 and 0x02500000-0x02600000 should be plenty.
+ * 0x02a00000-0x02b00000 and 0x02c00000-0x02d00000 should be plenty.
*
* On ARM, both the DTB and any possible initrd must be loaded such that they
* fit inside the lowmem mapping in Linux. In practice, this usually means not
* more than ~700M away from the start of the kernel image but this number can
* be larger OR smaller depending on e.g. the 'vmalloc=xxxM' command line
* parameter given to the kernel. So reserving memory from low to high
- * satisfies this constraint again. Reserving 1M at 0x02600000-0x02700000 for
- * the DTB leaves rest of the free RAM to the initrd starting at 0x02700000.
+ * satisfies this constraint again. Reserving 1M at 0x02c00000-0x02d00000 for
+ * the DTB leaves rest of the free RAM to the initrd starting at 0x02d00000.
* Even with the smallest possible CPU-GPU memory split of the CPU getting
- * only 64M, the remaining 25M starting at 0x02700000 should allow quite
+ * only 64M, the remaining 19M starting at 0x02d00000 should allow quite
* large initrds before they start colliding with U-Boot.
*/
#define ENV_MEM_LAYOUT_SETTINGS \
"fdt_high=" FDT_HIGH "\0" \
"initrd_high=" INITRD_HIGH "\0" \
"kernel_addr_r=0x00080000\0" \
- "scriptaddr=0x02400000\0" \
- "pxefile_addr_r=0x02500000\0" \
- "fdt_addr_r=0x02600000\0" \
- "ramdisk_addr_r=0x02700000\0"
+ "scriptaddr=0x02a00000\0" \
+ "pxefile_addr_r=0x02b00000\0" \
+ "fdt_addr_r=0x02c00000\0" \
+ "ramdisk_addr_r=0x02d00000\0"
#if CONFIG_IS_ENABLED(CMD_MMC)
#define BOOT_TARGET_MMC(func) \

Any ideas?

Regards,
Salvatore

[Vagrant Cascadian]

Adding u-boot maintainers for rpi (Matthias Brugger, Peter Robinson)
platforms and u-boot list to CC.

Post by Salvatore Bonaccorso
Thanks for tracking this down. I would like to loop in Masahiro and
upstream to see if something can/should be done on upstream side.
Masahiro, in short, upstream change 994b7ac1697b ("arm64: remove
special treatment for the link order of head.o") (which got backported
as well to 6.1.14) caused the vmlinuz size to icrease significantly,
causing boot failures on Raspberry Pi 3 Model B Plus with u-boot
parameters previously working. Full quoting the Debian report below

In general it would be nice to not have the kernel grow nearly 25% in
size from a single commit; was that an expected outcome from the above
upstream change? Was the "special treament" originally done to keep the
kernel size down?

As for u-boot, It seems u-boot might need to update the various load
addresses for the kernel, device tree and ramdisk at some point
regardless of weather this particular issue gets fixed in the kernel, as
the kernel will likely continue to grow a bit over time...

Aurelien Jarno included a patch referenced below which bumps the
tolerances in u-boot from 36MB to 42MB.

Post by Salvatore Bonaccorso

Following the upgrade of the kernel from 6.1.12-1 to 6.1.15-1 on a
Raspberry Pi 3 Model B Plus, u-boot (from the u-boot-rpi package) failed
| 40175552 bytes read in 1695 ms (23 MiB/s)
| 43794863 bytes read in 1817 ms (23 MiB/s)
| Moving Image from 0x80000 to 0x200000, end=2990000
| ERROR: RD image overlaps OS image (OS=0x200000..0x2990000)
I tracked the issue to a significant increase of the kernel size between
| 31492 /boot/vmlinuz-6.1.0-5-arm64
| 39236 /boot/vmlinuz-6.1.0-6-arm64
This is more than the 36MB that is allowed by u-boot with the default
load addresses. A workaround is to shift the load addresses at the
u-boot level as in the attached patch.
I have tracked issue on the upstream kernel side to the following commit
| commit 3e3e4d234d46e48480a7c7c35399fa811182e8ef
| Date: Thu Oct 13 08:35:00 2022 +0900
|
| arm64: remove special treatment for the link order of head.o
|
| commit 994b7ac1697b4581b7726d2ac64321e3c840229b upstream.
|
| In the previous discussion (see the Link tag), Ard pointed out that
| arm/arm64/kernel/head.o does not need any special treatment - the only
| piece that must appear right at the start of the binary image is the
| image header which is emitted into .head.text.
|
| The linker script does the right thing to do. The build system does
| not need to manipulate the link order of head.o.
|
The problem is still reproducible on Linus' master.
I am reporting the bug to the linux package as I believed there is no
real reason for such an increase in the kernel size. In case I missed
something and this is actually wanted, the bug can be reassigned to the
u-boot package.
Regards
Aurelien
--- u-boot-2023.01+dfsg.orig/include/configs/rpi.h
+++ u-boot-2023.01+dfsg/include/configs/rpi.h
@@ -95,32 +95,32 @@
* text_offset bytes (specified in the header of the Image) into a 2MB
* boundary. The 'booti' command relocates the image if necessary. Linux uses
* a default text_offset of 0x80000. In summary, loading at 0x80000
- * satisfies all these constraints and reserving memory up to 0x02400000
- * permits fairly large (roughly 36M) kernels.
+ * satisfies all these constraints and reserving memory up to 0x02a00000
+ * permits fairly large (roughly 42M) kernels.
*
* scriptaddr and pxefile_addr_r can be pretty much anywhere that doesn't
* conflict with something else. Reserving 1M for each of them at
- * 0x02400000-0x02500000 and 0x02500000-0x02600000 should be plenty.
+ * 0x02a00000-0x02b00000 and 0x02c00000-0x02d00000 should be plenty.
*
* On ARM, both the DTB and any possible initrd must be loaded such that they
* fit inside the lowmem mapping in Linux. In practice, this usually means not
* more than ~700M away from the start of the kernel image but this number can
* be larger OR smaller depending on e.g. the 'vmalloc=xxxM' command line
* parameter given to the kernel. So reserving memory from low to high
- * satisfies this constraint again. Reserving 1M at 0x02600000-0x02700000 for
- * the DTB leaves rest of the free RAM to the initrd starting at 0x02700000.
+ * satisfies this constraint again. Reserving 1M at 0x02c00000-0x02d00000 for
+ * the DTB leaves rest of the free RAM to the initrd starting at 0x02d00000.
* Even with the smallest possible CPU-GPU memory split of the CPU getting
- * only 64M, the remaining 25M starting at 0x02700000 should allow quite
+ * only 64M, the remaining 19M starting at 0x02d00000 should allow quite
* large initrds before they start colliding with U-Boot.
*/
#define ENV_MEM_LAYOUT_SETTINGS \
"fdt_high=" FDT_HIGH "\0" \
"initrd_high=" INITRD_HIGH "\0" \
"kernel_addr_r=0x00080000\0" \
- "scriptaddr=0x02400000\0" \
- "pxefile_addr_r=0x02500000\0" \
- "fdt_addr_r=0x02600000\0" \
- "ramdisk_addr_r=0x02700000\0"
+ "scriptaddr=0x02a00000\0" \
+ "pxefile_addr_r=0x02b00000\0" \
+ "fdt_addr_r=0x02c00000\0" \
+ "ramdisk_addr_r=0x02d00000\0"
#if CONFIG_IS_ENABLED(CMD_MMC)
#define BOOT_TARGET_MMC(func) \

Bumping from 36MB to 42MB seems address the issue at hand, although
would it make sense to bump even higher... or is that starting to hit
other limitations?


Thanks everyone!


live well,
vagrant

[Aurelien Jarno]

Post by Vagrant Cascadian
Adding u-boot maintainers for rpi (Matthias Brugger, Peter Robinson)
platforms and u-boot list to CC.

Post by Salvatore Bonaccorso
Thanks for tracking this down. I would like to loop in Masahiro and
upstream to see if something can/should be done on upstream side.
Masahiro, in short, upstream change 994b7ac1697b ("arm64: remove
special treatment for the link order of head.o") (which got backported
as well to 6.1.14) caused the vmlinuz size to icrease significantly,
causing boot failures on Raspberry Pi 3 Model B Plus with u-boot
parameters previously working. Full quoting the Debian report below

In general it would be nice to not have the kernel grow nearly 25% in
size from a single commit; was that an expected outcome from the above
upstream change? Was the "special treament" originally done to keep the
kernel size down?

This is currently being tracked [1], but the issue seems to be linked to
the version of the tools used in Debian, and the fact that we build our
kernels with BTF support, so the issue is likely to be difficult to find.

Post by Vagrant Cascadian
As for u-boot, It seems u-boot might need to update the various load
addresses for the kernel, device tree and ramdisk at some point
regardless of weather this particular issue gets fixed in the kernel, as
the kernel will likely continue to grow a bit over time...

Yes that's probably a sane thing to do, at least in Debian.

Post by Vagrant Cascadian
Aurelien Jarno included a patch referenced below which bumps the
tolerances in u-boot from 36MB to 42MB.

Post by Salvatore Bonaccorso

Following the upgrade of the kernel from 6.1.12-1 to 6.1.15-1 on a
Raspberry Pi 3 Model B Plus, u-boot (from the u-boot-rpi package) failed
| 40175552 bytes read in 1695 ms (23 MiB/s)
| 43794863 bytes read in 1817 ms (23 MiB/s)
| Moving Image from 0x80000 to 0x200000, end=2990000
| ERROR: RD image overlaps OS image (OS=0x200000..0x2990000)
I tracked the issue to a significant increase of the kernel size between
| 31492 /boot/vmlinuz-6.1.0-5-arm64
| 39236 /boot/vmlinuz-6.1.0-6-arm64
This is more than the 36MB that is allowed by u-boot with the default
load addresses. A workaround is to shift the load addresses at the
u-boot level as in the attached patch.
I have tracked issue on the upstream kernel side to the following commit
| commit 3e3e4d234d46e48480a7c7c35399fa811182e8ef
| Date: Thu Oct 13 08:35:00 2022 +0900
|
| arm64: remove special treatment for the link order of head.o
|
| commit 994b7ac1697b4581b7726d2ac64321e3c840229b upstream.
|
| In the previous discussion (see the Link tag), Ard pointed out that
| arm/arm64/kernel/head.o does not need any special treatment - the only
| piece that must appear right at the start of the binary image is the
| image header which is emitted into .head.text.
|
| The linker script does the right thing to do. The build system does
| not need to manipulate the link order of head.o.
|
The problem is still reproducible on Linus' master.
I am reporting the bug to the linux package as I believed there is no
real reason for such an increase in the kernel size. In case I missed
something and this is actually wanted, the bug can be reassigned to the
u-boot package.
Regards
Aurelien
--- u-boot-2023.01+dfsg.orig/include/configs/rpi.h
+++ u-boot-2023.01+dfsg/include/configs/rpi.h
@@ -95,32 +95,32 @@
* text_offset bytes (specified in the header of the Image) into a 2MB
* boundary. The 'booti' command relocates the image if necessary. Linux uses
* a default text_offset of 0x80000. In summary, loading at 0x80000
- * satisfies all these constraints and reserving memory up to 0x02400000
- * permits fairly large (roughly 36M) kernels.
+ * satisfies all these constraints and reserving memory up to 0x02a00000
+ * permits fairly large (roughly 42M) kernels.
*
* scriptaddr and pxefile_addr_r can be pretty much anywhere that doesn't
* conflict with something else. Reserving 1M for each of them at
- * 0x02400000-0x02500000 and 0x02500000-0x02600000 should be plenty.
+ * 0x02a00000-0x02b00000 and 0x02c00000-0x02d00000 should be plenty.
*
* On ARM, both the DTB and any possible initrd must be loaded such that they
* fit inside the lowmem mapping in Linux. In practice, this usually means not
* more than ~700M away from the start of the kernel image but this number can
* be larger OR smaller depending on e.g. the 'vmalloc=xxxM' command line
* parameter given to the kernel. So reserving memory from low to high
- * satisfies this constraint again. Reserving 1M at 0x02600000-0x02700000 for
- * the DTB leaves rest of the free RAM to the initrd starting at 0x02700000.
+ * satisfies this constraint again. Reserving 1M at 0x02c00000-0x02d00000 for
+ * the DTB leaves rest of the free RAM to the initrd starting at 0x02d00000.
* Even with the smallest possible CPU-GPU memory split of the CPU getting
- * only 64M, the remaining 25M starting at 0x02700000 should allow quite
+ * only 64M, the remaining 19M starting at 0x02d00000 should allow quite
* large initrds before they start colliding with U-Boot.
*/
#define ENV_MEM_LAYOUT_SETTINGS \
"fdt_high=" FDT_HIGH "\0" \
"initrd_high=" INITRD_HIGH "\0" \
"kernel_addr_r=0x00080000\0" \
- "scriptaddr=0x02400000\0" \
- "pxefile_addr_r=0x02500000\0" \
- "fdt_addr_r=0x02600000\0" \
- "ramdisk_addr_r=0x02700000\0"
+ "scriptaddr=0x02a00000\0" \
+ "pxefile_addr_r=0x02b00000\0" \
+ "fdt_addr_r=0x02c00000\0" \
+ "ramdisk_addr_r=0x02d00000\0"
#if CONFIG_IS_ENABLED(CMD_MMC)
#define BOOT_TARGET_MMC(func) \

Bumping from 36MB to 42MB seems address the issue at hand, although
would it make sense to bump even higher... or is that starting to hit
other limitations?

From the comment above the code, I understand that the main limitation
is when allocating most of the memory to the GPU, leaving only 64MB to
the CPU.
It seems possible to do that on many RPi:
- On 256MB RPi, the maximum gpu_mem value is 192
- On 512MB RPi, the maximum gpu_mem value is 448
- On >= 1GB RPi, the maximum gpu_mem value is 944

Aurelien