For best security, you should (a) firewall the port (as you have done) and (b) remove all the software (as Patrick said).
You might want to avoid removing the software, but you should at least stop the software from listening. It's wise to use in-depth security, so if one layer (the firewall) fails, then another layer will succeed. So, on systems with systemd, do this:
# as root, or with sudo
# stop the service
systemctl stop $service
# disable the service, so it doesn't run at startup
systemctl mask $service
If you don't have systemd, then do this instead. But, if this is Debian, you should really upgrade before it goes out of support in Spring 2018!
service $service stop
update-rc.d $service disable