Hello Nikhil,
there could be several reasons for this.
One I meet often especially when working remotely is that RTP streams cannot be established between the stations. Either because of Windows firewall settings or due to VPN profile setup.
Please, do mind that if you do NOT record such calls (CDT-2-CDT) on server (SSR), then the streams is being established directly between CDTs.
Do you have SSR enabled also for direct calls? If not, are those CDTs in same network segment? If yes, disable Windows firewall temporarily.
Regards,
Dawood.