Locking on Per CPU variable

1
votes

I am reading Linux Kernel Development by Robert Love and trying to understand the kernel synchronization mechanism. I am trying to understand some points related to the locking mechanism which is mentioned in the book as follows -

"some situations do not require a spin lock, but do need kernel preemption disabled.The most frequent of these situations is per-processor data. If the data is unique to each processor, there might be no need to protect it with a lock because only that one processor can access the data. If no spin locks are held, the kernel is pre-emptive, and it would be possible for a newly scheduled task to access this same variable.

Consequently, even if this were a uniprocessor computer, the variable could be accessed pseudo-concurrently by multiple processes. Normally, this variable would require a spin lock (to prevent true concurrency on multiprocessing machines). If this were a per- processor variable, however, it might not require a lock.To solve this, kernel preemption can be disabled via preempt_disable()"

So considering a multiprocessor system here - I understand that while per-cpu variable is being manipulated by the current process, another process may be scheduled due to SMP and try to manipulate the same per-cpu variable hence the preemption needs to be disabled as explained in the book. But one point I am not able to understand is that what if we only disable kernel preemption and the current process is trying to manipulate the per-cpu data and at the same time an interrupt occurred on the current processor and since the interrupt has not been disabled, CPU stops the current task and starts executing interrupt handler, Now this handler also want to manipulate the same per-cpu variable. So in this case the variable may ended up containing inconsistent data ?

So does that mean that the interrupt also needs to be disabled on the current processor if the per-cpu variable is also accessed from interrupt handler ?

Platform: Linux on x86

2
linuxlinux-kernelsynchronizationlocking

2 Answers

1
votes

So considering a multiprocessor system here - I understand that while per-cpu variable is being manipulated by the current process, another process may be scheduled due to SMP and try to manipulate the same per-cpu variable hence the preemption needs to be disabled as explained in the book.

I think the SMP is not playing any role here because variable is per cpu (not shared among processors). Another process may be sheduled because scheduler interrupt will came, that have no deal with SMP and may happen also on uniprocessor systems. To avoid this preemption disabling is used. For uniprocessor systems with CONFIG_PREEMPT_ENABLE linux kernel says spinlock/unlock = preemp disable/enable

So does that mean that the interrupt also needs to be disabled on the current processor if the per-cpu variable is also accessed from interrupt handler ?

Yes it is! And that also true for uniprocessor systems. In this case you should use irqsave API (or equivalents):

spin_lock_irqsave(&xxx_lock, flags);
... critical section here ..
spin_unlock_irqrestore(&xxx_lock, flags);

For uniprocessor kernel this spinlock will turn into local irq enable/disabling. In SMP systems the spinlock is actual spinlock implemented through cmpxchg + irq disabling/enabling.

0
votes

For multiprocessor system and to protect percpu variables, we can just disable preemption and do local_irq_save. This way we avoid taking the spinlock. Spinlock requires atomicity across multiple CPU's. With per cpu variables it shall not be required.

local_irq_save(flags);
preempt_disable();
-- Modify the percpu variable
preempt_enable(); 
local_irq_restore(flags);

With this code other CPUs can work in parallel. None of the CPU spins.

If we take spinlock then we modify an atomic variable also other CPU comes then it has to wait/spin if spinlock is acquired.