When install Portal Server, you can ignore the error from Test Connection.
Because at that moment, the install process has not got the whole java environment setting.
You can let the install finishes.
When tomcat starts, it'll generate several logs. one of them, cmrserver.log, is in
Powerdesigner Portal Server\log
Other in Powerdesigner Portal Server\Tomcat\log
These logs will tell you what is going on.
Sysam error "WinSock:Connection refused" occurs if the license server is on 64bit Windows
and both IPv4 and IPv6 are enabled.
By default, sysam uses IPv4, so you can try disable IPv6.