Check the that the hash length is shorter than the message length. This
avoids:

    ./tools/../lib/rsa/rsa-verify.c:275:11: warning:
    ‘*db’ may be used uninitialized [-Wmaybe-uninitialized]
      275 |         db[0] &= 0xff >> leftmost_bits;

Fixes: 061daa0b61f0 ("rsa: add support of padding pss")
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

Tighten up symbol dependencies in a number of places.  Ensure that a SPL
specific option has at least a direct dependency on SPL.  In places
where it's clear that we depend on something more specific, use that
dependency instead.  This means in a very small number of places we can
drop redundant dependencies.

Reported-by: Pali Rohár <pali@kernel.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

Modify function to support any salt length instead of max
length only. Function now detects salt length by parsing
the content of db buffer. Note that it works with (but is
not limited to) zero-length, digest-length and max-length

Signed-off-by: SESA644425 <gioja.hermann@non.se.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

Prior to introduction of modifications in rsassa_pss functions
related to padding verification, doing a pass to reduce memory
consumption of function by replacing memory copies of parts of
const buffer by pointers to the original buffer (masked_db and
h are subparts of msg buffer which is declared const, salt is a
subpart of db which is a working buffer, unmodified after being
filled). New pointers scope is limited to the function where
they are declared (not returned to caller by any mean), zeroing
risk of memory fault related to the change.

Signed-off-by: SESA644425 <gioja.hermann@non.se.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

Prior to introduction of modifications in rsassa_pss functions
related to padding verification, doing a pass to update
const-correctness in targeted functions to comply with
coding-rules and avoid const-cast

Signed-off-by: SESA644425 <gioja.hermann@non.se.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

This commit adds the option SPL_RSA_VERIFY_WITH_PKEY.

Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Philippe Reynes <philippe.reynes@softathome.com>

Since OpenSSL 1.1.0, EVP_MD_CTX_create() is EVP_MD_CTX_new()
                     EVP_MD_CTX_destroy() is EVP_MD_CTX_free()
                     EVP_MD_CTX_init() is EVP_MD_CTX_reset()

As there's no need to reset a newly created EVP_MD_CTX, moreover
EVP_DigestSignInit() does the reset, thus call to EVP_MD_CTX_init()
can be dropped.
As there's no need to reset an EVP_MD_CTX before it's destroyed,
as it will be reset by EVP_MD_CTX_free(), call to EVP_MD_CTX_reset()
is not needed and can be dropped.

Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>

Add to support rsa 3072 bits algorithm in tools
for image sign at host side and adds rsa 3072 bits
verification in the image binary.

Add test case in vboot for sha384 with rsa3072 algorithm testing.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

It is useful to know where the verification data was written. Update the
API to return this.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add some more debugging to make it easier to see what is being tried and
what fails. Fix a few comment styles while here.

Signed-off-by: Simon Glass <sjg@chromium.org>