I’m not familiar with SAML, and as far as I know, the proxy directive doesn’t create a SERVER_PORT variable, but there is a server port - but the {port} placeholder is the the client’s port’s as connected to the server, probably not quite what you intended, which is why setting it to 443 fixes it, since you wanted the server port. (There isn’t a placeholder for the server port because that is known by the person who creates the config.)
That sounds correct … in the end the example python3-saml library wasn’t set up to observe X-Forwarded parameters out of the box. After I picked it apart it was essentially a configuration touch up.