Suricata crash

[kgleason]

This may be related to #1233, I'm not sure since I am running headless, and I don't have a functioning serial console to watch the thing boot.

When I attempt to start suricata from the UI, nothing happens. The service doesn't start, and there is no error reported. Odd. The scheduler is still downloading all of the rules updates.

I ssh'd in, and tried to start suricata as root from the CLI.

root@OPNsense:/usr/local/etc/rc.d # service suricata start
Starting suricata.
29/10/2016 -- 17:16:47 - <Info> - Including configuration file installed_rules.yaml.
Illegal instruction (core dumped)
/usr/local/etc/rc.d/suricata: WARNING: failed to start suricata

Total bummer. None of my configs have been edited by hand -- all configuration was done via the OPNSense UI.

Here are some bits from dmesg that may be helpful:

CPU: AMD Athlon(tm) 64 Processor 3700+ (2411.78-MHz K8-class CPU) 
  Origin="AuthenticAMD"  Id=0xf4a  Family=0xf  Model=0x4  Stepping=10
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow!+,3DNow!>
real memory  = 3221225472 (3072 MB)
avail memory = 3078905856 (2936 MB)

I can grab any additional logs that you'd like to see.

Also from dmesg, I see a lot of the following:

MCA: Bank 4, Status 0xa40000000005001b 
MCA: Global Cap 0x0000000000000105, Status 0x0000000000000000
MCA: Vendor "AuthenticAMD", ID 0xf4a, APIC ID 0
MCA: CPU 0 UNCOR GTLB LG error
MCA: Address 0x4a59040

I did find this issue where it seems like something similar to this was happening on Xeon (and since I'm on some OLD AMD hardware, it seems like it might be relevant). It seems like it is a much older version of suricata however, so I'm not 100% convinced that it is relevant.

I don't know what what point suricata stopped running. It's been at least 7 days, based on what I can see from the log files -- all of the /var/log/suricata.log files are essentially empty, and all of the files in /var/log/suricata/ are actually empty.

Please let me know what I can do to assist.

[fabianfrz]

Illegal Instruction is an error coming from your CPU. It means that the compiler used instructions which are not available on your CPU. Without knowing what is responsible for that it will be hard to find that out. It should work if you compile it by yourself as a workaround for now.

[fichtner]

Hi Kirk,

We ran into this last year indeed, Suricata builds enables native extensions on compile by default, topic here:

https://redmine.openinfosecfoundation.org/issues/373

This, however, was brought to OPNsense in June and to FreeBSD in September 2015:

freebsd/freebsd-ports@5153b42906a710296

Crashes on AMD Athlon 64 bit CPUs seem to be prevalent, circling around the ET rule set. Some users noted that disable those rules will make Suricata not crash. We're still compiling hints as to why this happens, because so far nothing obvious sticks out.

A on-box recompile should fix this, using i386 instead of amd64 likely works, too. It would be helpful to get confirmation on either of these theories as we're missing the hardware to reproduce under lab conditions.

Thanks,
Franco

[kgleason]

Franco,
I disabled all of my rules, and I get the same result when starting Suricata.
It doesn't seem like there is a ports tree by default in OPNSense. Can I just install the ports, and use that to compile?

[fichtner]

Hi Kirk,

Here's the sequence from the top of my head (we do have a ports tree):

# opnsense-code tools ports
# cd /etc
# ln -s /usr/tools/config/16.7/make.conf /etc/make.conf
# cd /usr/ports/security/suricata
# make all deinstall install PRODUCT_FLAVOUR=OpenSSL

(If you use LibreSSL switch the flavour setting)

[fichtner]

(forgot that we have opnsense-code utility now...)

[fichtner]

@kgleason I could not find an issue with Suricata per se, but can you try this package instead (using 16.7.7 just to make sure)?

# pkg add -f https://pkg.opnsense.org/snapshots/suricata-3.1.2.txz

[fichtner]

Your CPU lacks SSE3 support, which is a requirement for Hyperscan:

intel/hyperscan#20

The whole situation is a real pickle, because Suricata loads the library and then things just go boom because I don't know why. It's a "portable unless not portable" approach I need to see how to get out of... :(

Do you use the Hyperscan setting?

[kgleason]

I started the recompile from ports before I left for work (using make all deinstall install PRODUCT_FLAVOR=LibreSSL). Came home and tried starting Suricata to the same error. I'll give the pkg a try this evening. Assuming that I should make deinstall on the port before I try the pkg?

I am on 16.7.7

[kgleason]

That pkg seems to have worked. Here is what I did:

make deinstall #(from the port)

rm -rf /usr/local/etc/suricata

pkg add -f https://pkg.opnsense.org/snapshots/suricata-3.1.2.txz

When I attempted to service suricata start I got an error about eth2. Odd. I went into the OPNSense UI, and added a couple of non-ET rulesets, and hit apply. Suricata was started immediately, but it seems to have broken all of my OB DNS. I still have to dig into a bit more.

Currently Suricata is turned off. I'll poke at it more this evening.

[kgleason]

After some additional prodding, Suricata is up and running, with rules loaded. Things seem to be working quite well.

It looks like that pkg I installed was from a snapshot. Do I need to keep an eye out for updates to suricata in the immediate future?

[fichtner]

Hi Kirk,

Thanks for confirming. I'm not entirely sure how to fix this other than recommending to use i386 images for this hardware, because that's the only difference: not having hyperscan included. Having a pseudo amd64-ish build that requires SSE3 is a bit hypocritical. At least it shouldn't run the instructions if you haven't even selected it as an engine.

I will, of course, provide updated builds of suricata without hyperscan, maybe figure out how to install this via a plugin. I will also try to get Suricata devs to look at this, maybe we can find a solution there, too.

For now, you can make sure your suricata stays in place by locking it from the firmware GUI packages page, or typing:

# pkg lock suricata

That will prevent it from updating via the GUI, but you will have to make sure you're manually reapplying the newer package if errors arise with the previous package.

Cheers,
Franco

[kgleason]

Franco,
Thanks. Sounds like this might be my excuse to buy a better system to run as my firewall. ;-)