network: bridge: iptables seems not to support comments

[haai]

Nomad version

Nomad v0.12.4 (8efaee4ba5e9727ab323aaba2ac91c2d7b572d84)

Operating system and Environment details

Linux srv 4.4.59+ #25426 SMP PREEMPT Wed Jul 8 03:21:29 CST 2020 x86_64 GNU/Linux synology_apollolake_918+

iptables version is v1.6.0

Issue

I have here a specific problem. I am trying to run nomad scheduler on my synology NAS and it seems that the iptables does not support comments. The bridge has been created and also other ip tables and rules but on the below rule an error is thrown in the job.

Reproduction steps

Having a synology disk station :) or iptables without comments extension and creating job with

network { mode = "bridge" }

Job file (if appropriate)

Nomad Client logs (if appropriate)

failed to setup alloc: pre-run hook "network" failed:
 failed to configure networking for alloc:
 failed to configure network:
 running [/sbin/iptables -t nat -C CNI-fdde69e63278bfb5db76bed4 -d 172.26.64.2/20 -j ACCEPT -m comment --comment 
 name: "nomad" id: "75c4cec6-9774-5f3d-e4ef-bde96b4860dc" --wait]:
 exit status 2:
 iptables v1.6.0: Couldn't load match `comment':No such file or directory Try `iptables -h' or 'iptables --help' for more information.

Nomad Server logs (if appropriate)

[tgross]

Hello @haai! That's a bit of a surprise. It looks like from the log message here that the comment is being added in the CNI library. So we'll need to open an issue with upstream for that.

That being said, can you run ls /lib/modules/$(uname -r)/kernel/net/netfilter/ for me? I worry that if your iptables doesn't have the comments extension that you might not have the extensions that the CNI library needs to run network isolation at all.

[haai]

hi @tgross. thanks for your reply. Yes thats right .. I was curious also about this and checked the cni plugins repository. The plugins set the comments. But nomad passes the comments to the plugins also with UUIDs. Here for example: https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/bridge.go#L615

                if isLayer3 && n.IPMasq {
		chain := utils.FormatChainName(n.Name, args.ContainerID)
		comment := utils.FormatComment(n.Name, args.ContainerID)
		for _, ipn := range ipnets {
			if err := ip.TeardownIPMasq(ipn, chain, comment); err != nil {
				return err
			}
		}
	}

I was thinking, when the comments are not being added by the plugin will nomad then continue to work or are these UUIDs in the comment does have any meaning, for bookkeeping e.g.?

Unfortunately the exact command does not work on my node. They have all modules in just /lib/modules. Its a kinda long list :(

$ sudo ls /lib/modules/
8021q.ko	     btusb.ko		     dib7000m.ko		      dvb-usb-m920x.ko		 hd29l2.ko	      leds-lp3943.ko	    nf_conntrack_pptp.ko       rt2800lib.ko	   sp8870.ko		       tda9887.ko	     videobuf-dvb.ko
8812au.ko	     button.ko		     dib7000p.ko		      dvb-usb-mxl111sf.ko	 hdpvr.ko	      lg2160.ko		    nf_conntrack_proto_gre.ko  rt2800usb.ko	   sp887x.ko		       tea5761.ko	     videobuf-vmalloc.ko
a8293.ko	     carl9170.ko	     dib8000.ko			      dvb-usb-nova-t-usb2.ko	 hfsplus.ko	      lgdt3305.ko	    nf_defrag_ipv4.ko	       rt2x00lib.ko	   stb0899.ko		       tea5767.ko	     videodev.ko
ablk_helper.ko	     cbc.ko		     dib9000.ko			      dvb-usb-opera.ko		 hid-generic.ko       lgdt3306a.ko	    nf_defrag_ipv6.ko	       rt2x00usb.ko	   stb6000.ko		       tm6000.ko	     video.ko
acpi-cpufreq.ko      ccm.ko		     dibx000_common.ko		      dvb-usb-pctv452e.ko	 hid.ko		      lgdt330x.ko	    nf_nat_ipv4.ko	       rtl2830.ko	   stb6100.ko		       ts2020.ko	     vxlan.ko
aesni-intel.ko	     cdc-acm.ko		     dm-bufio.ko		      dvb-usb-rtl28xxu.ko	 hidp.ko	      lgs8gl5.ko	    nf_nat.ko		       rtl2832.ko	   stp.ko		       ttpci-eeprom.ko	     wm8775.ko
aes-x86_64.ko	     cdc_ether.ko	     dm-flakey.ko		      dvb-usb-technisat-usb2.ko  hmac.ko	      lgs8gxx.ko	    nf_nat_masquerade_ipv4.ko  rtl8187.ko	   stv0288.ko		       ttusbdecfe.ko	     xc4000.ko
af9013.ko	     cfbcopyarea.ko	     dm-snapshot.ko		      dvb-usb-ttusb2.ko		 horus3a.ko	      libiscsi.ko	    nf_nat_pptp.ko	       rtl8192c-common.ko  stv0297.ko		       ttusb_dec.ko	     xc5000.ko
af9033.ko	     cfbfillrect.ko	     drbd.ko			      dvb-usb-umt-010.ko	 i2c-algo-bit.ko      libiscsi_tcp.ko	    nf_nat_proto_gre.ko        rtl8192cu.ko	   stv0299.ko		       tua6100.ko	     xfrm4_mode_beet.ko
af_key.ko	     cfbimgblt.ko	     drm_kms_helper.ko		      dvb_usb_v2.ko		 i915.ko	      llc.ko		    nf_nat_redirect.ko	       rtl_usb.ko	   stv0367.ko		       tua9001.ko	     xfrm4_mode_transport.ko
ah4.ko		     cfg80211.ko	     drm.ko			      dvb-usb-vp702x.ko		 igb.ko		      lnbh25.ko		    nfnetlink.ko	       rtlwifi.ko	   stv0900.ko		       tuner.ko		     xfrm4_mode_tunnel.ko
ah6.ko		     cifs.ko		     drm_panel_orientation_quirks.ko  dvb-usb-vp7045.ko		 iosf_mbi.ko	      lnbp21.ko		    nfnetlink_queue.ko	       s5h1409.ko	   stv090x.ko		       tuner-simple.ko	     xfrm4_tunnel.ko
ansi_cprng.ko	     cls_fw.ko		     drx39xyj.ko		      e1000e.ko			 ip6table_filter.ko   lnbp22.ko		    nfsd.ko		       s5h1411.ko	   stv6110.ko		       tuner-types.ko	     xfrm6_mode_beet.ko
appletalk.ko	     cls_u32.ko		     drxd.ko			      e4000.ko			 ip6table_mangle.ko   loop.ko		    n_hdlc.ko		       s5h1420.ko	   stv6110x.ko		       tuner-xc2028.ko	     xfrm6_mode_transport.ko
arc4.ko		     compat_xtables.ko	     drxk.ko			      ec100.ko			 ip6_tables.ko	      lrw.ko		    nxt200x.ko		       s5h1432.ko	   synoacl_vfs.ko	       tun.ko		     xfrm6_mode_tunnel.ko
as102_fe.ko	     cpufreq_performance.ko  ds3000.ko			      ecb.ko			 ip6_udp_tunnel.ko    m88ds3103.ko	    nxt6000.ko		       s921.ko		   synobios.ko		       tunnel4.ko	     xfrm6_tunnel.ko
ascot2e.ko	     cpufreq_powersave.ko    dvb-as102.ko		      echainiv.ko		 ipcomp6.ko	      m88rs2000.ko	    openvswitch.ko	       saa7115.ko	   syno_extent_pool.ko	       tunnel6.ko	     xfrm_algo.ko
atbm8830.ko	     cpufreq_stats.ko	     dvb-core.ko		      ecryptfs.ko		 ipcomp.ko	      m88rs6000t.ko	    option.ko		       sch_htb.ko	   syno_flashcache_control.ko  tveeprom.ko	     xfrm_ipcomp.ko
ath3k.ko	     crc32c-intel.ko	     dvb-pll.ko			      eeprom_93cx6.ko		 ip_set_hash_ip.ko    mac80211.ko	    or51132.ko		       sch_netem.ko	   syno_hddmon.ko	       udf.ko		     xfrm_user.ko
ath9k_common.ko      crc-ccitt.ko	     dvb-ttusb-budget.ko	      ehci-hcd.ko		 ip_set.ko	      macvlan.ko	    or51211.ko		       sch_sfq.ko	   synotty.ko		       udp_tunnel.ko	     xhci-hcd.ko
ath9k_htc.ko	     crc-itu-t.ko	     dvb-usb-a800.ko		      ehci-pci.ko		 iptable_filter.ko    max2165.ko	    p8022.ko		       seqiv.ko		   syscopyarea.ko	       uhci-hcd.ko	     xhci-pci.ko
ath9k_hw.ko	     cryptd.ko		     dvb-usb-af9005.ko		      em28xx-dvb.ko		 iptable_mangle.ko    mb86a16.ko	    pci-stub.ko		       sg.ko		   sysfillrect.ko	       usb-common.ko	     x_tables.ko
ath9k.ko	     cs53l32a.ko	     dvb-usb-af9015.ko		      em28xx.ko			 iptable_nat.ko       mb86a20s.ko	    phy_alloc_0815_x64.ko      sha256_generic.ko   sysimgblt.ko		       usbcore.ko	     xt_addrtype.ko
ath.ko		     ctr.ko		     dvb-usb-af9035.ko		      esp4.ko			 ip_tables.ko	      mc44s803.ko	    ppp_async.ko	       si2157.ko	   target_core_ep.ko	       usbhid.ko	     xt_conntrack.ko
au0828.ko	     cts.ko		     dvb-usb-anysee.ko		      esp6.ko			 ipt_MASQUERADE.ko    md4.ko		    ppp_deflate.ko	       si2165.ko	   target_core_file.ko	       usbip-core.ko	     xt_geoip.ko
au8522_common.ko     cx22700.ko		     dvb-usb-au6610.ko		      etxhci-hcd.ko		 ip_tunnel.ko	      md5.ko		    ppp_generic.ko	       si2168.ko	   target_core_iblock.ko       usbip-host.ko	     xt_iprange.ko
au8522_decoder.ko    cx22702.ko		     dvb-usb-az6007.ko		      exfat.ko			 ipv6.ko	      mpls_gso.ko	    ppp_mppe.ko		       si21xx.ko	   target_core_mod.ko	       usblp.ko		     xt_ipvs.ko
au8522_dig.ko	     cx231xx-dvb.ko	     dvb-usb-az6027.ko		      exportfs.ko		 ip_vs.ko	      msi001.ko		    pppoe.ko		       sierra.ko	   target_core_multi_file.ko   usbnet.ko	     xt_limit.ko
aufs.ko		     cx231xx.ko		     dvb-usb-ce6230.ko		      fat.ko			 ip_vs_rr.ko	      msp3400.ko	    pppox.ko		       sit.ko		   tc90522.ko		       usbserial.ko	     xt_LOG.ko
authencesn.ko	     cx2341x.ko		     dvb-usb-cinergyT2.ko	      fbdev.ko			 ir-kbd-i2c.ko	      mt2060.ko		    ppp_synctty.ko	       slhc.ko		   tcm_loop.ko		       usb-storage.ko	     xt_mac.ko
authenc.ko	     cx24110.ko		     dvb-usb-cxusb.ko		      fb.ko			 irqbypass.ko	      mt2063.ko		    pptp.ko		       smsdvb.ko	   tda10021.ko		       usbvision.ko	     xt_mark.ko
b2c2-flexcop.ko      cx24113.ko		     dvb-usb-dib0700.ko		      fb_sys_fops.ko		 iscsi_target_mod.ko  mt20xx.ko		    processor.ko	       smsmdtv.ko	   tda10023.ko		       usb_wwan.ko	     xt_multiport.ko
b2c2-flexcop-usb.ko  cx24116.ko		     dvb-usb-dibusb-common.ko	      fc0011.ko			 iscsi_tcp.ko	      mt2131.ko		    psnap.ko		       smsusb.ko	   tda10048.ko		       v4l2-common.ko	     xt_nat.ko
backlight.ko	     cx24117.ko		     dvb-usb-dibusb-mb.ko	      fc0012.ko			 iscsi_trgt.ko	      mt2266.ko		    pvrusb2.ko		       snd-hwdep.ko	   tda1004x.ko		       v4l2-dv-timings.ko    xt_NFQUEUE.ko
bcm203x.ko	     cx24120.ko		     dvb-usb-dibusb-mc.ko	      fc0013.ko			 isl6405.ko	      mt312.ko		    qm1d1c0042.ko	       snd.ko		   tda10071.ko		       ves1820.ko	     xt_recent.ko
bcm3510.ko	     cx24123.ko		     dvb-usb-digitv.ko		      fc2580.ko			 isl6421.ko	      mt352.ko		    qt1010.ko		       snd-mixer-oss.ko    tda10086.ko		       ves1x93.ko	     xt_REDIRECT.ko
bfusb.ko	     cx25840.ko		     dvb-usb-dtt200u.ko		      flashcache.ko		 isl6423.ko	      mt7601u.ko	    quota_tree.ko	       snd-pcm.ko	   tda18212.ko		       veth.ko		     xt_set.ko
bluetooth.ko	     cxd2820r.ko	     dvb-usb-dtv5100.ko		      flashcache_syno.ko	 isofs.ko	      mxl111sf-demod.ko     quota_v2.ko		       snd-pcm-oss.ko	   tda18218.ko		       vfat.ko		     xt_state.ko
bonding.ko	     cxd2841er.ko	     dvb-usb-dvbsky.ko		      ftdi_sio.ko		 it913x.ko	      mxl111sf-tuner.ko     r8168.ko		       snd-rawmidi.ko	   tda18271c2dd.ko	       vhost.ko		     xt_TCPMSS.ko
bridge.ko	     cypress_firmware.ko     dvb-usb-dw2102.ko		      fuse.ko			 itd1000.ko	      mxl301rf.ko	    r820t.ko		       snd-seq-device.ko   tda18271.ko		       vhost_net.ko	     xt_tcpudp.ko
br_netfilter.ko      deflate.ko		     dvb-usb-ec168.ko		      generic_bl.ko		 ix2505v.ko	      mxl5005s.ko	    r8712u.ko		       snd-timer.ko	   tda665x.ko		       vhost_scsi.ko	     zd1211rw.ko
bsd_comp.ko	     des_generic.ko	     dvb-usb-friio.ko		      gf128mul.ko		 kvm-intel.ko	      mxl5007t.ko	    rc-core.ko		       snd-usb-audio.ko    tda8083.ko		       videobuf2-core.ko     zl10036.ko
btbcm.ko	     dib0070.ko		     dvb-usb-gl861.ko		      ghash-generic.ko		 kvm.ko		      netconsole.ko	    regmap-i2c.ko	       snd-usb-hiface.ko   tda8261.ko		       videobuf2-memops.ko   zl10039.ko
btintel.ko	     dib0090.ko		     dvb-usb-gp8psk.ko		      glue_helper.ko		 l2tp_core.ko	      nf_conntrack_ipv4.ko  rfcomm.ko		       snd-usbmidi-lib.ko  tda826x.ko		       videobuf2-v4l2.ko     zl10353.ko
btrfs.ko	     dib3000mb.ko	     dvb-usb.ko			      gre.ko			 l2tp_ppp.ko	      nf_conntrack_ipv6.ko  rodsp_ep.ko		       soundcore.ko	   tda827x.ko		       videobuf2-vmalloc.ko  zram.ko
btrtl.ko	     dib3000mc.ko	     dvb-usb-lmedm04.ko		      hci_uart.ko		 l64781.ko	      nf_conntrack.ko	    rpcsec_gss_krb5.ko	       sp2.ko		   tda8290.ko		       videobuf-core.ko

[tgross]

But nomad passes the comments to the plugins also with UUIDs.

Yes, but that's expected by the interface we're calling into.

I was thinking, when the comments are not being added by the plugin will nomad then continue to work or are these UUIDs in the comment does have any meaning, for bookkeeping

Right, the snippet you linked to is a good example of what they're used for: after the allocation has finished, we need to be able to clean up the iptables entry and the only way to "ID" an entry is via the comment.

Unfortunately the exact command does not work on my node. They have all modules in just /lib/modules

Ok, the nf_* and xt_* group are the ones that we're mostly interested in for network isolation. The host you have is missing a lot of the modules that, for example, an Ubuntu VM I have lying around here has... but I'll be honest and say I'm not sure exactly which ones are relevant to network isolation in Nomad without some more research.

This is definitely an unusual kernel build for a server; I might not build a server with a preemptible kernel but presumably the Synology folks had some specific in mind. You may want to try using host networking first to verify that everything else is working. But at that point I think you're going to need to add kernel modules to get bridge networking working; I'm not sure how feasible that is with this kind of appliance.

[haai]

Thx @tgross for your answer :)
Ok then I guess I have bad cards here. Either I got that iptables feature into the upstream of the distro or I compile the kernel myself. But thats not really a good option.
The fact that nomad uses the ids makes the case for patching the plugins also not applicable.
That means that probably more features are missing -.-
Yes I am using host networking alreay and it works up to now. docker driver is also usable but docker creates own bridge. but for consul connect I need the nomad bridge networking with cni plugins right?

[tgross]

Ok then I guess I have bad cards here. Either I got that iptables feature into the upstream of the distro or I compile the kernel myself. But thats not really a good option.

It's definitely not ideal. Fortunately because this is Linux the appliance vendor should be making their kernel tree available, and your kernel build config should be available at something like /boot/config-4.4.59-##-blah-blah to use as a starting point. But I totally get the trepidation there.

(And we probably should have some documentation around what minimal set of kernel features is required for bridge networking support, but I'm not totally sure about how stable those requirements are.)

but for consul connect I need the nomad bridge networking with cni plugins right?

That's my understanding. I'm going to tag @nickethier and @shoenig just to double-check me, but I don't think there's a way around that.

[shoenig]

To use Consul Connect with sidecar proxies, yes CNI plugins + bridge networking is required.

You could make use of Consul Connect Native to get around that requirement, but that requires using a connect library in your app, which may not be practical or possible.

[tgross]

I'm going to close this as it's a matter of dependencies on the kernel build.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.