| Age | Commit message (Collapse) | Author |
|---|
|
Commit 37eaca4ab33d ("lib: sbi_ipi: Return error for invalid hartids")
added a weight check against the HSM-interruptible mask, which excludes
harts in STOPPED state. This causes sbi_ipi_send_many() to return
SBI_EINVAL when the kernel sends a remote fence (e.g. TLB shootdown)
whose hmask includes a hart that was stopped by CPU hotplug:
__sbi_rfence_v02_call: hbase = [0] hmask = [0x3] failed (error [-22])
Validate requested hartids against the domain's assigned hartmask
instead. This still rejects truly invalid or cross-domain hartids, but
allows HSM-STOPPED harts to be silently skipped — which is safe because
a stopped hart will do a full local TLB flush on its next HSM start
before re-entering S-mode.
Fixes: 37eaca4ab33d ("lib: sbi_ipi: Return error for invalid hartids")
Signed-off-by: Chen Pei <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Anup Patel <[email protected]>
|
|
A platform may contain multiple IPI devices. In certain use cases,
such as power management, it may be necessary to send an IPI through a
specific device to wake up a CPU. For example, if an IMSIC is powered
down and reset, the core cannot receive IPIs from it, so the wake-up must
instead be triggered through the CLINT.
Suggested-by: Anup Patel <[email protected]>
Signed-off-by: Nick Hu <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Anup Patel <[email protected]>
|
|
The platform specfic IPI init is not need anymore because using
IPI device rating multiple IPI devices can be registered in any
order as part of the platform specific early init.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Samuel Holland <[email protected]>
Tested-by: Nick Hu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Anup Patel <[email protected]>
|
|
A platform can have multiple IPI devices (such as ACLINT MSWI,
AIA IMSIC, etc). Currently, OpenSBI rely on platform calling
the sbi_ipi_set_device() function in correct order and prefer
the first avaiable IPI device which is fragile.
Instead of the above, introduce IPI device rating and prefer
the highest rated IPI device. This further allows extending
the sbi_ipi_raw_clear() to clear all available IPI devices.
Signed-off-by: Anup Patel <[email protected]>
Tested-by: Nick Hu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Anup Patel <[email protected]>
|
|
sbi_send_ipi() should return SBI_ERR_INVALID_PARAM if even one hartid
constructed from hart_mask_base and hart_mask, is not valid.
Signed-off-by: Andrew Jones <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Anup Patel <[email protected]>
|
|
Now that the SBI IPI core clears IPIs at warm boot in a generic way,
none of the drivers or platforms use these hooks, and we can remove
them. Platforms need only to initialize the driver once during cold
init. If other hooks are needed in the future, they can be added to
struct sbi_ipi_device.
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
sbi_ipi_init() expects the platform warm init function to clear IPIs
on the local hart, but there is already a generic function to do this.
After this change, none of the existing drivers need a warm init
callback.
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
All existing users of this operation target the current hart, and it
seems unlikely that a future user will need to clear the pending IPI
status of a remote hart. Simplify the logic by changing .ipi_clear (and
its wrapper sbi_ipi_raw_clear()) to always operate on the current hart.
This incidentally fixes a bug introduced in commit 78c667b6fc07 ("lib:
sbi: Prefer hartindex over hartid in IPI framework"), which changed the
.ipi_clear parameter from a hartid to a hart index, but failed to update
the warm_init functions to match.
Fixes: 78c667b6fc07 ("lib: sbi: Prefer hartindex over hartid in IPI framework")
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
This removes several hartid/hartindex conversions, as well as two loops
through the mask for broadcast IPIs.
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
Use the IPI .update callback to exclude the local hart. This allows
reusing the normal logic for broadcasting an IPI to all active harts.
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
This avoids calls to the expensive sbi_hartid_to_hartindex() function
and also makes the firmware smaller.
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
sbi_ipi_event_create() disallows registering an IPI event with a NULL
.process callback, so the function pointer will never be NULL here.
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Reviewed-by: Xiang W <[email protected]>
|
|
An IPI sent to the local hart can be processed directly instead of
triggering the IPI device. This is more efficient, and it avoids a
deadlock when the .sync callback is defined. Since interrupts are
disabled while handling an ecall, the IPI would not get delivered
until the next mret, but sbi_ipi_sync() is called before then.
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Reviewed-by: Xiang W <[email protected]>
|
|
Currently, failures in sbi_ipi_send() are silently ignored, which makes
them difficult to debug. Instead, abort sending the IPI and pass back
the error, but still synchronize any IPIs already sent.
Signed-off-by: Samuel Holland <[email protected]>
Signed-off-by: Anup Patel <[email protected]>
|
|
Multiple harts may try to send IPI to a particular target hart A
in which case the send_ipi() should be called only when the old
value of the hart A ipi_type is zero.
Signed-off-by: Xiang W <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
Currently, there are no barriers before or after the ipi_clear()
device callback which forces ipi_clear() device callback to always
use non-relaxed MMIO writes.
Instead of above, we use wmb() in after the ipi_clear() device
callback which pairs with the wmb() done before the ipi_send()
device callback. This also allows device ipi_clear() callback
to use relaxed MMIO writes.
Signed-off-by: Anup Patel <[email protected]>
Reported-by: Bo Gan <[email protected]>
|
|
Currently, we have a smp_wmb() between atomic_raw_set_bit() and
ipi_send() device callback whereas the MMIO writes done by the
device ipi_send() callback will also include a barrier.
We can avoid unnecessary/redundant barriers described above by
allowing relaxed MMIO writes in device ipi_send() callback. To
achieve this, we simply use wmb() instead of smp_wmb() before
calling device ipi_send().
Signed-off-by: Anup Patel <[email protected]>
Reported-by: Bo Gan <[email protected]>
|
|
Let us prefer hartindex over hartid in IPI framework which in-turn
forces IPI users to also prefer hartindex.
Signed-off-by: Anup Patel <[email protected]>
|
|
Currently, the sbi_hartmask is indexed by hartid which puts a
limit on hartid to be less than SBI_HARTMASK_MAX_BITS.
We extend the sbi_hartmask implementation to use hartindex and
support updating sbi_hartmask using hartid. This removes the
limit on hartid and existing code works largely unmodified.
Signed-off-by: Xiang W <[email protected]>
Signed-off-by: Anup Patel <[email protected]>
|
|
The previous definition had the assumption that the machine word length
is equal to the word length of LONG. Remove this assumption and add a
static check to prevent errors in subsequent modifications.
Signed-off-by: Xiang W <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
Simplify sbi_ipi_process() by removing goto statement.
Signed-off-by: Xiang W <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
We have redundant semicolon at quite a few places so let's remove it.
Signed-off-by: Xiang W <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
When tlb_fifo is full, it will wait and affect the ipi update to
other harts. This patch is optimized.
Signed-off-by: Xiang W <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Tested-by: Anup Patel <[email protected]>
|
|
The original sbi_ipi will be processed by hart by hart, after optimization,
send ipi first and finally wait together.
Signed-off-by: Xiang W <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Tested-by: Anup Patel <[email protected]>
|
|
Since commit 50d4fde1c5a4 ("lib: Remove redundant sbi_platform_ipi_clear()
calls"), the IPI sent from the boot hart in wake_coldboot_harts() is not
cleared in the secondary harts until they reach sbi_ipi_init(). However,
sbi_hsm_init() and sbi_hsm_hart_wait() are called earlier, so a secondary
hart might enter sbi_hsm_hart_wait() with an already pending IPI.
sbi_hsm_hart_wait() makes sure the hart leaves the loop only when it is
actually ready, so a pending unrelated IPI should not cause safety issues.
However, it might be inefficient on certain hardware, because it prevents
"wfi" from stalling the hart even if the hardware supports this, making the
hart needlessly spin in a "busy-wait" loop.
This behaviour can be observed, for example, in a QEMU VM (QEMU 7.2.0) with
"-machine virt" running a Linux guest. Inserting delays in
sbi_hsm_hart_start() allows reproducing the issue more reliably.
The comment in wait_for_coldboot() suggests that the initial IPI is needed
in the warm resume path, so let us clear it before init_warm_startup()
only.
To do this, sbi_ipi_raw_clear() was created similar to sbi_ipi_raw_send().
Signed-off-by: Evgenii Shatokhin <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
C language standard uses true/false for the boolean type.
Let's switch to that for better language compatibility.
Signed-off-by: Bin Meng <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
Reviewed-by: Samuel Holland <[email protected]>
Tested-by: Samuel Holland <[email protected]>
|
|
If the ecall SBI_EXT_HSM_HART_START is called it might try to wake the
secondary hart using sbi_ipi_raw_send() to send an IPI to the hart.
This can fail if there is no IPI device but no error is returned from
sbi_ipi_raw_send() so the ecall returns as if the action completed and
the caller continues without noticing (in the case of Linux it just hangs
waiting for the secondary hart to become active)
Fix this by changing sbi_ipi_raw_send() to return and error, and if an
error is returned, then return it via SBI_EXT_HSM_HART_START call.
Signed-off-by: Ben Dooks <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
RISC-V SBI v0.3 specification defines a set of firmware events that can
provide additional information about the current firmware context. All
of the firmware event monitoring are enabled now. The firmware
events must be defined as raw perf event with MSB set as specified in the
specification.
Reviewed-by: Anup Patel <[email protected]>
Signed-off-by: Atish Patra <[email protected]>
|
|
The parameter owner of function sbi_scratch_alloc_offset() is never used.
The scratch memory is small. We should not use it for debug information in
future. Hence eliminate the parameter.
Signed-off-by: Heinrich Schuchardt <[email protected]>
Reviewed-by: Xiang W <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
Instead of having ipi_send() and ipi_clear() callbacks in
platform operations, it will be much simpler for ipi driver
to directly register these operations as a device to sbi_ipi
implementation.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Alistair Francis <[email protected]>
|
|
The sbi_platform_ipi_clear() called from wait_for_coldboot() and
sbi_hsm_hart_wait() is redundant because IPI will be automatically
cleared by sbi_platform_ipi_init() called from sbi_ipi_init().
Further, wait_for_coldboot() is common for warm startup and warm
resume path so the sbi_platform_ipi_clear() called in warm resume
path cause resuming HART to miss an IPI injected other HART to
wakeup the HART.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
A hart can take interrupt in the new HSM states introduced by the
SBI HSM suspend function (such as SUSPENDED state) so we rename
sbi_hsm_hart_started_mask() to something more generic such as
sbi_hsm_hart_interruptible_mask().
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
The sbi_hsm_hart_started_mask() API should take one more parameter
to allow caller specify domain under which started_mask is being
generated. Further, the sbi_hsm_hart_started_mask() depends on
sbi_hsm_hart_get_state() which also should return HART state under
specified domain.
This patch updates both sbi_hsm_hart_started_mask() and
sbi_hsm_hart_get_state() as-per above.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Alistair Francis <[email protected]>
|
|
This patch removes scratch parameter from most sbi_ipi functions.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
The scratch parameter in sbi_hsm_hart_started_mask() API is now
redundant hence removing it.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Bin Meng <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
The sbi_hartid_to_scratch() API can fail for non-existent HARTs so
all uses of sbi_hartid_to_scratch() API should check return value.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Bin Meng <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
This patch simplify HART id to scratch macro as follows:
1. Remove current "scratch" pointer argument because now we
use HART id to scratch table
2. Rename sbi_hart_id_to_scratch() to sbi_hartid_to_scratch()
to have macro name consistent with the name of callback
in struct sbi_scratch
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Bin Meng <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
The sbi_current_hartid() being a regular function is quite
expensive because for callers it is a function call instead
of a direct CSR read. This patch converts sbi_current_hartid()
into a macro in riscv_asm.h.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
This patch replaces use of sbi_hart_available_mask() API with
sbi_hsm_hart_started_mask() API.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Bin Meng <[email protected]>
|
|
This patch adds support for hart hotplug in OpenSBI using a generic WFI
based approach. Hart hotplug can be achieved via SBI HSM extension which
allows supervisor mode software to start or stop any harts anytime.
Any platform wishes to implement platform specific hart hotplug must
implement both hart_start and hart_stop in addition to enable platform
feature SBI_PLATFORM_HAS_HART_HOTPLUG.
Signed-off-by: Atish Patra <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
We don't need to separately call sbi_ipi_send() for current HART
in sbi_ipi_send_many(). Instead, we can simplify the for-loop in
sbi_ipi_send_many() and call sbi_ipi_send() for all HARTs in the
for-loop itself.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Bin Meng <[email protected]>
|
|
As per the latest SBI specification, all online harts should receive
IPI if hbase is set to -1.
Set the target mask to all available hart mask if hbase is -1.
Signed-off-by: Atish Patra <[email protected]>
Reviewed-by: Anup Patel <[email protected]>
|
|
Currently, the remote TLB management is initialized via IPI init
which is counter intuitive. This patch initializes remote TLB
management directly from init_coldboot() and init_warmboot()
after IPI init is done.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
This patch drops _fifo from the name of various sbi_tlb_fifo_xyz()
functions because all these functions deal with remote TLB managment
and FIFO is the per-HART data structure used internally by remote
TLB implementation.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
This patch introduces sbi_ipi_event_create/destroy() APIs and
struct sbi_ipi_event_ops for creating/destroying IPI events
at runtime based of event operations.
This new APIs will help platform code and utils code to create
custom IPI events which are not part of generic OpenSBI library.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
Instead of directly calling sbi_ipi_send_many(), we introduce
sbi_ipi_send_halt() for halting a set of HARTs.
This way in future we can assign any IPI event number for HART
halting within sbi_ipi.c only.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
Instead of directly calling sbi_ipi_send_many(), we introduce
sbi_ipi_send_smode() for injecting S-mode software interrupts.
This way in future we can assign any IPI event number for S-mode
IPIs within sbi_ipi.c only.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
The struct sbi_ipi_data is only used in sbi_ipi.c so move it
to sbi_ipi.c from sbi_ipi.h.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
Instead of doing sbi_hang() we should do sbi_exit() upon
halt IPI.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
|
We add sbi_ipi_exit() API for exit path handling in sbi_exit()
implementation.
Signed-off-by: Anup Patel <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
|
