Can you enable interrupts in page fault handler? Is there an ARM kernel contention with preemptive scheduling?
I got an ARM kernel oops in UDP receiving code with CONFIG_PREEMPT, or when interrupt is enabled in fault handler.
The problem is similar to what another user reported here. But in my case when I send 110% load UDP packets to the system (system drops about 10% packets), kernel oops in a few minutes. This happens only if there are some busybox shell scripts running, not if only the UDP receiving program is running. I've tracked the data addresses it always looks good, the buffer was allocated and used before it is freed.
There are two ways to avoid it:
[1] When changing scheduling from preempt (CONFIG_PREEMPT) to preempt_voluntary, the problem goes away. Is this a known issue with ARM on kernel 2.6.39? With preempt scheduling I also see problem in jffs2 after a long while, but not with preempt_voluntary.
For a moment I suspected it is the Ethernet DMA fully utilized the bus thus blocking CPU from loading its TLB entry thus causing page fault. I'm deducing because busybox scripts need to be in the picture, when a script is spawned it creates address space and load many TLB entries thus overloading the bus. If preempt_voluntary is a solution, can DMA blocking bus be ruled out?
The test I'm running is a LTIB kernel 2.6.39.4 lpclinux on a phy3250 based system.
[2] Some more tests showed that the page fault handler is nested by Ethernet interrupts. When disabling interrupts in the kernel page fault handler __dabt_svc, but keep it enabled in the user page fault handler __dabt_user, the problem goes away. If not, the nest level goes up to 4 and it oops'ed. So the question is: Is enabling interrupts in page fault handler correct?
The test code for [2] goes below. Lines with @@@@ are added or modified. Then capture the nesting level in do_DataAbort().
file arch/arm/kernel/entry-armv.S:
__dabt_svc:
svc_entry
... ...
@
@ set desired IRQ state, then call main handler
@
debug_entry r1
@@@@Not_Enable_Irq_In_Dabtsvc
ldr r2, =armv_dabtsvc_count @@@@
ldr r3, [r2] @@@@
add r3, r3, #1 @@@@
str r3, [r2] @@@@
msr cpsr_c, r9 @@@@disable thisk
mov r2, r2 @@@@add this extra inst
mov r2, sp
bl do_DataAbort
@
@ IRQs off again before pulling preserved data off the stack
@
disable_irq_notrace
ldr r2, =armv_dabtsvc_count @@@@
ldr r3, [r2] @@@@
sub r3, r3, #1 @@@@
str r3, [r2] @@@@
@
@ restore SPSR and restart the instruction
@
ldr r2, [sp, #S_PSR]
svc_exit r2 @ return from exception
UNWIND(.fnend )
ENDPROC(__dabt_svc)
And add the variable to the file too:
file arch/arm/kernel/entry-armv.S:
@@@@save nesting level:
.data @@@@
.align @@@@
armv_dabtsvc_count: @@@@
.long 0 @ count svc entry @@@@
I'm trying to link all these up. Can kernel experts see whether all the tests make sense? Is disabling interrupts in page fault handler is a valid solution?
Edit: The oops in page fault handler is not the first failure. There was a "do_bad_area" in a proceeding alignment handler. Subsequently that failed fixup to unaligned access caused the page fault. Yes as someone commented below, fixing unaligned access is very troublesome. Those unaligned accesses are from ip_input, ip_fragment, and udp stack. Once I fixed all those in the stack, the problem is gone.
Edit again: The problem is with two operations in alignment handler: It fetches the instruction, and fetches data the instruction refers to. The oops is reported by data access, but the cause is fetching instruction failed with a first page fault failure. Since the fetch instruction is in kernel space, the page is always valid, that indicates a silicon bug. If change the code to fetch again it would succeed, that confirms it is more likely a silicon bug. Interrupt gets into the picture because of excess TLB flushing it brings in. For short, TLB loading is automatic thus fetching instruction in kernel space cannot fail. But still it failed.
copy_from_user(). There is no reason to block interrupts in this case; you increase latency. Probably you are masking the problem with all of your suggestions; the kernel code is full of subtleties. – artless noiseusermode, to prevent a mix-up in the alignment trap flags. The page fault handlers should not callcopy_from_user(); they are part of the implementation.__get_user()is different, it is trying to get the code/instruction that caused the alignment trap. Theswitchstatement manual decodes the instruction. – artless noise