Your router could have settings in place for its NAT causing this to happen - what you want is a 1:1 NAT or a static NAT (terminology may differ depending on router)
