FNE not working anymore with openssl v3

[TrinityTonic]

We've recently updated our ubuntu version from ubuntu20 --> ubuntu22. Whereas ubuntu20 used openssl version v1, ubuntu22 uses openssl version v3. We noticed that we get an error during compilation of the FlexNet Embedded library, particularly inside the openssl glue code (FlxExtGlueOpenSsl.c).

The error during compilation:

/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/jenkins/agent/deps/flx2go/public/lib/linux-gcc-aarch64/release/libflx2go.a(FlxExtGlueOpenSsl.c.o): in function `zed6130b851': FlxExtGlueOpenSsl.c:(.text+0x8a0): undefined reference to `EVP_PKEY_size' collect2: error: ld returned 1 exit status

We checked what has changed and apparently openssl v3 does not contain the requested function "EVP_PKEY_size" anymore which is why compilation of the openssl glue code fails. I've looked into the latest release of your porting toolkit (2022.09), however it seems that it still relies on that function (and hence on openssl v1). 

It is already "known" and apparently deliberate that the function was removed from openssl v3 - seee https://answers.launchpad.net/ubuntu/+question/703323.

Have you been aware of this? In fact this would make the FNE code unusable with newest openssl versions. Could you provide ideas / suggestions to overcome this problem? We would not like to switch to another crypto lib (mbedtls / wolfssl) at this point though.

[mjolnir · Accepted solution]

We do build against OpenSSL 3.0.X internally, so it should be fine for you. I believe you're seeing a linker error rather than a compilation error as, although the function has been removed, they have provided a compatibility macro detailed here: https://www.openssl.org/docs/man3.0/man3/EVP_PKEY_size.html

If you recompile FlxExtGlueOpenSsl.c, making sure to include the OpenSSL 3.0 headers when you do so, you should not see that linker error.

Hope that helps!