HTTP/1.1 200 OK Date: Thu, 09 Oct 2025 19:57:38 GMT Server: Apache Last-Modified: Fri, 17 May 2024 12:32:53 GMT ETag: "e4c1-618a58ce8c740-gzip" Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN Strict-Transport-Security: max-age=10886400; includeSubDomains; preload Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data: *.youtube.com *.openstreetmap.org *.vimeo.com *.frikanalen.no *.nuug.no yewtu.be *.kjemi.uio.no *.oreilly.com *.skolelinux.de *.googleapis.com remarkjs.com *.gstatic.com api.flattr.com; img-src 'self' twitter-badges.s3.amazonaws.com nuug.no data: secure.gravatar.com blob:; script-src-elem 'self' 'unsafe-inline' yewtu.be remarkjs.com *.flattr.com digg.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' yewtu.be blob: *.googleapis.com *.gstatic.com; style-src 'self' 'unsafe-inline' *.googleapis.com; connect-src 'self' *.nuug.no *.googleapis.com; worker-src 'self' blob:; frame-src 'self' *.youtube.com blob:; frame-ancestors 'self' nuug.no; report-uri /csp-report-endpoint; Referrer-Policy: same-origin Report-To: {"group":"default","max_age":31536000,"endpoints":[{"url":"https://f24dd70b087e791fbf1c81a30b0bb7f2.report-uri.com/a/d/g"}],"include_subdomains":true} Access-Control-Allow-Origin: nuug.no Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Headers: Content-Type Content-Length: 20061 Content-Type: text/html; charset=UTF-8 class: center, middle # Network Management with the OpenBSD Packet Filter Toolset ## EuroBSDCon 2023 #### Coimbra, September 14th, 2023 ### Peter Hansteen, Massimiliano Stucchi, Tom Smyth Updated version to be found at [https://nxdomain.no/~peter/pf_fullday.pdf](https://nxdomain.no/~peter/pf_fullday.pdf) --- ## whoami (who am I), part Peter: ### Peter Hansteen <peter@bsdly.net> - Sysadmin, [OpenBSD](https://www.openbsd.org) user since before the millenium - Wrote [The Book of PF](https://www.nostarch.com/pf3), now in its third edition - Blog at [bsdly.blogspot.com](https://bsdly.blogspot.com) about (lack of) sanity in IT - Works at [Tietoevry Create](https://tietoevry.com/en/create) - Yes, I'll do another book any decade now --- ## whoami (who am I), part Max: ### Massimiliano Stucchi - Technical Advisor at The Internet Society - Here representing myself only - IPv6 "Enthusiast" - https://stucchi.ch - @stucchimax@social.secret-wg.org --- ## whoami (who am I), part Tom: ### Tom Smyth - working in IT since 2000 - CTO wireless Connect Ltd. an ISP in Ireland - Opinions are mine and may be my companies also :) - PF student, an avid reader of the Book of PF. - I really Enjoy networking with OpenBSD - Maintainer of the NSH network Shell for OpenBSD. --- ## Introduce yourself - A quick introduction about yourself: - Your name - Your favourite BSD - Your experience with networking - Your experience with PF - Your goal(s) ??? This introduction serves for us to understand the level of the room, and decide on how to better suit the tutorial. Getting an idea of the level makes it so that we might quickly go through the basics, and focus on something more advanced, rather than spend time on something that everybody knows already. --- ## Agenda 1. PF Basics 2. Exercise: Host configuration 3. NAT and Redirects 4. Exercise: Setup a gateway 5. Hosting Services 6. Exercise: Hosting Services, redirects 7. Traffic Shaping 8. Exercise: Setting up queueing 9. Tips 10. Troubleshooting 11. Exercise: NAT64 12. End --- class: center, middle # PF Basics ### Section 1 --- ## It all started with a copyright violation - OpenBSD up to 2.9 (2001) used Darren Reed's IPFilter - Almost, but not quite BSD licensed - No right to distribute changed versions - IPFilter removed on May 29th, 2001 - First commit of the new PF code June 24 2001 at 19:48:58 UTC - OpenBSD 3.0 release pushed to Dec 1 by the extra effort - *License audit* of src tree + ports tree followed But let's jump to the present - ??? --- ## Network design 101 How to build a maintainable network: - Start with something simple - Think dual stack (IPv6 and IPv4) - Users - Access to local and remote services - and their security requirements - The services you offer and the ones you consume - i.e: SMTP, WWW, DNS - **Keep it simple, not stupid** --- ## Know what a Firewall can (and cannot) do! A firewall can: - allow certain packets flowing to or through the firewall device - drop certain packets flowing to or through the firewall device - redirect or NAT packets according to policy There are limits though - Inbound interface bandwith limitations - CPU I/O interface driver packets per second(PPS) limitations A firewall cannot - block inbound Flood Denial of Service attacks that exceed: - the inbound interface capacity - the CPU I/O / interupt capacity of the interface Any software firewall can be overwhelmed with high packet rate Denial of Service attacks. If the scale of the attack exceeds the capability of the firewall --- ## But first, how do we build rulesets? The basic _rule_ format is, **verb** _criteria_ **actions** ... options ```shell pass in on egress proto tcp to egress port ssh ``` ```shell match out on egress nat-to egress ``` ```shell block all ``` You can do complete rulesets with only these, but you may also want to declare things -> ( **egress** ? -> the _interface group_ consisting of interfaces that have a default route ) --- ## Next, declarations Declaring objects your rules will use: **Macros:** ```shell macro-name = "definition" ``` **Tables:** ```shell table <tablename> ``` **Queues:** ```shell queue queuename on interface bandwidth number ... ``` You can also *set* options. We'll be returning to all of those, in the meantime [man pf.conf](https://man.openbsd.org/pf.conf) is always your friend. --- ## Block by default - A firewall is supposed to block by default, right? - Let's start with the ruleset for a client - Useful baseline: - block by default - pass traffic we generate ```shell block pass from (self) ``` --- ## In practice - On my laptop in my home network, that expands to: ```shell $ doas pfctl -vnf /etc/pf.conf block drop all pass inet6 from ::1 to any flags S/SA pass on lo0 inet6 from fe80::1 to any flags S/SA pass on iwm0 inet6 from fe80::a2a8:cdff:fe63:abb9 to any flags S/SA pass inet6 from 2001:470:28:658:a2a8:cdff:fe63:abb9 to any flags S/SA pass inet6 from 2001:470:28:658:8c43:4c81:e110:9d83 to any flags S/SA pass inet from 127.0.0.1 to any flags S/SA pass inet from 192.168.103.126 to any flags S/SA ``` ??? It would be nice in this slide to ask questions, to see the general knowledge about protocols: - IPv6 LLA - IPv6 GUA - IPv6 GUA with privacy extensions - IPv4 Private address This can give us an idea of how to proceed with IPv6 depending on the level of the participants. --- ## Anatomy of a ruleset - Ruleset evaluated top to bottom - **Last match wins** (unless _quick_ is used) - Stateful by default - For more details, see [man pf.conf](https://man.openbsd.org/pf.conf) - https://man.openbsd.org/pf.conf **RULE order matters**, keep in mind when you write your own later --- ## A ruleset can contain - **Options** - General configuration - **Macros** - Content to be expanded - **Tables** - When macros fall short (for IP addresses only) - **Redirections** - **Rules** - How PF should behave - The order of the items in the ruleset used to be important, but not anymore --- ## Options - General configuration for pf - Useful for debugging, applying default timeout values, etc. ```shell set limit states 100000 ``` - or ```shell set debug debug set loginterface dc0 set timeout tcp.first 120 set timeout tcp.established 86400 set timeout { adaptive.start 6000, adaptive.end 12000 } ``` --- ## Macros - Content that will later be expanded in context - Name must start with a _letter_, _digit_, or _underscore_ - Useful to keep rulesets simple and clean ```shell ext_if = "kue0" all_ifs = "{ $ext_if lo0 }" pass out on $ext_if from any to any pass in on $ext_if proto tcp from any to any port 25 ``` - Could also be port or address ranges ```shell bacula_ports = "9101:9103" dmz_hosts = "192.0.2.1 - 192.0.2.254" ``` ??? If we have more slide space, then separate the second example in its own slide --- ## Tables - When IP address list macros grow too long, use tables instead - Hold only _addresses_ and _networks_ - No port ranges - Provide fast lookups ```shell table <bruteforce> persist counters ``` - They can be used in rules ```shell block from <bruteforce> ``` - Unlike macros with port or address lists, _do not_ expand to multiple rules - More optimised ruleset --- ## Tables continued - Content can be loaded from files or from CLI ```shell table <popflooders> persist counters file "/home/peter/popflooders" ``` useful if you save contents to preserve across reboots, such as ```shell $ doas pfctl -t popflooders -T show >/home/peter/popflooders ``` possibly in [cron](https://man.openbsd.org/cron) jobs - use [pfctl](https://man.openbsd.org/pfctl) to add/remove entries from command line ```shell $ doas pfctl -t bruteforce -T add 192.0.2.11 2001:db8::dead:beef:baad:f00d ``` - optionally maintained by daemons such as [spamd](https://man.openbsd.org/spamd), [bgpd](https://man.openbsd.org/bgpd) and [dhcpd](https://man.openbsd.org/dhcpd) --- class: center, middle ## Implementing your spec ### How would you do this in your home/soho infrastructure ? ??? We ask the participants how they think they would implement this in their infrastructure. This is to create discussion, and hopefully make it so that we all learn a bit by trial-and-error. Let's put out the question, wait for some time to see if anyone comes up with a nice description, and then walk them through what is a sensible setup. This is in preparation for the first exercise which comes next. --- class: center, middle # Exercise 1 ### Protecting your host --- ## Excercise 1 - Let's start - Lab environment: - Open your favourite browser, then - Go to [labs.pftutorial.net](https://labs.pftutorial.net) - Username is "labX" (where X is your assigned number) - Password is "pftutorial" --- ## Exercise 1 - First steps - Get to the console for __fwlabX__ - Check that pf is indeed loaded and running (*hint*: [pfctl](https://man.openbsd.org/pfctl)) - Wait until everyone has connected in order to proceed --- ## Exercise 1 - net config - Configure the external interface on gateway - _vi /etc/hostname.vio0_ ```shell inet 10.255.255.XX/24 !route add 0/0 10.255.255.254 inet6 fd18:b5d:cafe::XX/64 !route add -inet6 2000::/3 fd18:b5d:cafe::a !route add -inet6 fd00::/8 fd18:b5d:cafe::a ``` - and then _vi /etc/resolv.conf_ ```shell nameserver 10.255.255.254 nameserver fd18:b5d:cafe::a ``` followed by ```shell sh /etc/netstart ``` --- ## Exercise 1 - on gateway - Start with a block ruleset ```shell block pass quick inet6 proto tcp from fd18::/16 to port ssh pass quick inet6 proto icmp6 from fd18::/16 ``` - Allow traffic to be generated from your host, and allow ICMPv6 ```shell pass from self ``` and then, reload _pf.conf_ ```shell pfctl -vnf /etc/pf.conf pfctl -f /etc/pf.conf ``` - **NB:** Reload pf this way after every statement in the exercises --- ## Exercise 1 - Tests - From your gateway ping a host - First IPv6 ```shell # ping6 fd18:b5d:cafe::a PING fd18:b5d:cafe::a (fd18:b5d:cafe::a): 56 data bytes 64 bytes from fd18:b5d:cafe::a: icmp_seq=0 hlim=64 time=0.548 ms 64 bytes from fd18:b5d:cafe::a: icmp_seq=1 hlim=64 time=0.492 ms 64 bytes from fd18:b5d:cafe::a: icmp_seq=2 hlim=64 time=0.494 ms ``` - Then IPv4 ```shell # ping stucchi.ch PING stucchi.ch (45.129.224.40): 56 data bytes 64 bytes from 45.129.224.40: icmp_seq=0 ttl=56 time=6.264 ms 64 bytes from 45.129.224.40: icmp_seq=1 ttl=56 time=6.273 ms 64 bytes from 45.129.224.40: icmp_seq=2 ttl=56 time=6.117 ms ``` --- ## Exercise 1 - Wrap up - Does ping work? - Do other commands work? - working from total block, proceed to make restricted workstation - name resolution - _http_ and _https_ - Access public web sites, other Internet resources. - What would it take to access the other lab hosts? --- class: center, middle # Questions ? ??? Let's ask if there are any questions before continuing. Make sure we have everyone onboard. --- class: center, middle # Gateway, NAT and Redirects ### Section 2 --- ## Easy! How do we improve on this? - Make a 'firewall': - a point of policy enforcement - a gateway - filter for other hosts - redirection tricks --- ## Introducing NAT - **N**etwork **A**ddress **T**ranslation ([RFC1631](https://tools.ietf.org/html/rfc1631) onwards) - -> 'Hide' several hosts behind 1 or more public addresses, using [RFC1918](https://tools.ietf.org/html/rfc1918) addresses - -> can be used by ISPs for conserving scarce IP addresses in large networks (CG-NAT) 100.64.0.0/10 - Modern PF has _nat-to_ on 'pass' and 'match' rules: ```shell match out on $extif inet nat-to ($extif) ``` - *Neat trick*: egress is the interface group that has a default route, you can filter on it ```shell match out on egress inet nat-to (egress) ``` - In modern networks we **should** (also) have IPv6 (inet6) ??? NAT, the stopgap measure that's old enough to drink, more than 22 years old. NAT was created as a temporary measure that hasn't been replaced by now. Not even IPv6. We haven't discusses "egress" yet, so this is the right time to introduce it. Spend some time discussing it, along with the rest of the NAT specifications. Unfortunately, there's also NAT for IPv6, called NAT66 --- ## A (filtering) Gateway _"I decide which packets pass"_ #### Enable forwarding: - Temporarily set from command line with [sysctl](https://man.openbsd.org/sysctl): ```shell # sysctl net.inet.ip.forwarding=1 # sysctl net.inet6.ip6.forwarding=1 ``` - Make permanent in [/etc/sysctl.conf](https://man.openbsd.org/sysctl.conf) ```shell net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1 ``` --- ## The minimal gateway - Do you *NAT* for IPv4? Of course you do. - Do you run IPv6? Of course you do. ```shell ext_if=bge0 int_if=bge1 match out on egress inet nat-to ($ext_if) block all pass proto tcp from { self, $int_if:network } ``` - The "pass" rule, withouth _inet_ or _inet6_ applies to both **Keep in mind**: This is a point of policy enforcement --- ## A Point of policy enforcement - Now some policy, and macros ```shell ext_if=bge0 int_if=bge1 client_out = "{ ftp-data, ftp, ssh, domain, pop3, auth, nntp, http, \ https, 2628, 5999, 8000, 8080 }" udp_services = "{ domain, ntp }" match out on egress inet nat-to ($ext_if) * block * pass quick proto { tcp, udp } to port $udp_services keep state * * pass proto tcp from $int_if:network to port $client_out * * pass proto tcp to self port ssh ``` - What services do your clients consume? ??? Log to on the system we have for showing out, and then show the rules there and how they expand to different parts. --- ## Letting dhcpd(8) direct access OpenBSD [dhcpd(8)](https://man.openbsd.org/dhcpd) can interact with your ruleset via tables: */etc/rc.conf.local* ```shell dhcpd_flags="-L leased_ip_table -A abandoned_ip_table -C changed_ip_table bge1" ``` ```shell ext_if=bge0 int_if=bge1 * table <abandoned_ip_table> persist counters * table <changed_ip_table> persist counters * table <leased_ip_table> persist counters client_out = "{ ftp-data, ftp, ssh, domain, pop3, auth, nntp, http, \ https, 2628, 5999, 8000, 8080 }" udp_services = "{ domain, ntp }" match out on egress inet nat-to ($ext_if) block pass quick proto { tcp, udp } to port $udp_services keep state * pass proto tcp from <leased_ip_table> to port $client_out pass proto tcp to self port ssh ``` **=>** only pass traffic from hosts with active leases from *me* ??? Maybe move this slide to a later section. --- ## Redirects (and divert-to) Modern PF has two classes of redirect * **rdr-to** on match and pass rules - rewrite destination address while filtering (locally or even to other hosts) ```shell pass in on egress to port www rdr-to $webserver ``` * **divert-to** on match and pass rules - [divert()](https://man.openbsd.org/divert) socket for local use ```shell pass in on egress to port smtp divert-to 127.0.0.1 port spamd ``` --- ## FTP Proxy - If your users need to access FTP services, [ftp-proxy](https://man.openbsd.org/ftp-proxy) is what you need - FTP does not easily pass through a block firewall, some help is needed ```shell $ doas rcctl enable ftpproxy6 ``` - or for IPv4 ```shell $ doas rcctl enable ftpproxy ``` - and then add an anchor and divert rules to your config ```shell anchor "ftp-proxy/*" ... pass in quick inet proto tcp to port ftp divert-to 127.0.0.1 port 8021 pass in quick inet6 proto tcp to port ftp divert-to ::1 port 8021 pass out proto tcp from $proxy to port ftp ``` There is even a reverse mode (**-R**) for when you host FTP servers, see [man ftp-proxy](https://man.openbsd.org/ftp-proxy) --- class: center, middle # Exercise 2 ### Protecting your network --- ## Exercise 2 - Goals - Your network grows, you become a gateway - Extend the configuration to enable the network to access the internet --- background-image: url(images/exercise2.png) ## Exercise 2 - Your network --- ## Exercise 2 - Turn on ip forwarding (sysctl) ```shell # sysctl net.inet.ip.forwarding=1 # sysctl net.inet6.ip6.forwarding=1 ``` - Set up NAT ```shell match out on egress inet nat-to (egress) ``` Also, pass traffic from that local net --- ## Exercise 2 - preparation - Configure the hosts with the following IPv6 addresses - *Gateway (vio1):* fd18:b5d:XX::a/64 - *Host1:* fd18:b5d:XX::80/64 - *Host2:* fd18:b5d:XX::25/64 <br/ > <br/> - On Host1 and Host2, set fd18:b5d:XX::a as the default IPv6 gateway - and also the following IPv4 addresses - *Gateway (vio1):* 192.168.XX.1/24 - *Host1:* 192.168.XX.2/24 - *Host2:* 192.168.XX.3/24 - On Host1 and Host2 set 192.168.XX.1 as the default IPv4 gateway --- ## Exercise 2 - check your results - From client 1, ping a host on the internet - First IPv6 ```shell # ping6 stucchi.ch PING stucchi.ch (2001:41d0:8:6ed8::80): 56 data bytes 64 bytes from 2001:41d0:8:6ed8::80: icmp_seq=1 hlim=56 time=7.414 ms 64 bytes from 2001:41d0:8:6ed8::80: icmp_seq=2 hlim=56 time=6.333 ms 64 bytes from 2001:41d0:8:6ed8::80: icmp_seq=3 hlim=56 time=6.441 ms ``` - Then IPv4 ```shell # ping stucchi.ch PING stucchi.ch (37.59.51.141): 56 data bytes 64 bytes from 37.59.51.141: icmp_seq=0 ttl=56 time=6.264 ms 64 bytes from 37.59.51.141: icmp_seq=1 ttl=56 time=6.273 ms 64 bytes from 37.59.51.141: icmp_seq=2 ttl=56 time=6.117 ms ``` --- # Exercise 2b: FTP Try fetching *ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest* ```shell # wget ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest ``` Check your result If it didn't work, configure FTP-proxy and try again. --- class: center, middle # Hosting Services ### Section 3 --- ## Hosting services behind your gateway - Now you actually want some 'pass in on egress' rules :) - Get your specifications clear, put in writing: - your services and the ports they use - the names could be in **/etc/services** already - the hosts (IP addresses) that run the services - decide who/what/where to be reachable for/from - Check services requiring extra help (i.e. proxying). --- ## Proxies and other helpers - OpenBSD comes with several proxies and other service helper programs in the base system: - [ftp-proxy](https://man.openbsd.org/ftp-proxy) (you guessed it) - [relayd](https://man.openbsd.org/relayd) (load balancing and lots more) - [spamd](https://man.openbsd.org/spamd) (if you run SMTP - annoy spammers) <br><br><br> - There are also such things as squid and varnish (web proxies) in packages <br><br><br> - Clients and services in the same subnet? Or do the DMZs? --- ## DMZ, defined ### '*D*e-*M*ilitarized *Z*one' - when a group of host needs special treatment <br><br> - attached to separate interfaces(s), separate sub-rulesets <br><br> - And **YES**, you can have several <br><br> - Think multiple customers, *N* environments each --- background-image: url(images/exercise3.png) class: right ## DMZ, illustrated --- ##Allowing some services in ```shell ext_if=bge0 # adjust to what your system has int_if=bge1 # adjust to what your system has client_out = "{ ftp-data, ftp, ssh, domain, pop3, auth, nntp, http, \ https, 2628, 5999, 8000, 8080 }" udp_services = "{ domain, ntp }" * webserver = "192.0.2.227" * webports = "{ http, https }" * emailserver = "192.0.2.225" * email = "{ smtp, pop3, imap, imap3, imaps, pop3s }" * nameservers = "{ 192.0.2.221, 192.0.2.223 }" match out on egress inet nat-to ($ext_if) block pass quick proto { tcp, udp } to port $udp_services keep state pass proto tcp from $int_if:network to port $client_out pass proto tcp to self port ssh * pass proto tcp to $webserver port $webports * pass proto tcp to $emailserver port $email * pass log proto tcp from $emailserver to port smtp * pass inet proto { tcp, udp } to $nameservers port domain ``` Now try loading this with *pfctl -vnf /etc/pf.conf* and see what this expands to (you can also fetch [pf.services01.conf](samples/pf.services01.conf) and try) --- ## Tackling noise (attacks) with state-tracking options ### Scenario: ssh bruteforcers In our previous ruleset, add ```shell table <bruteforce> persist counters block from <bruteforce> ``` and change the **ssh** rule to ```shell * pass proto tcp to port ssh flags S/SA keep state \ * (max-src-conn 15, max-src-conn-rate 2/10, overload <bruteforce> flush global) ``` - Tune to taste - More info on options in [man pf.conf](https://man.openbsd.org/pf.conf) - Remember *[pfctl](https://man.openbsd.org/pfctl) expire* ??? max-src-conn - see man page but max number of simultaneous connections from one host max-src-conn-rate - number of new connections per number of seconds --- ## Tackling noise (attacks) with state-tracking options ### Scenario: Wordpress site - Wordpress is a usual target for many different attacks - -> Make the attackers suffer by forcing them through smaller "windows" - Use tables and block by connection rates - Mix and match settings, consult [man pf.conf](https://man.openbsd.org/pf.conf) and remember *[pfctl](https://man.openbsd.org/pfctl) expire* - Also see [Forcing the password gropers through a smaller hole with OpenBSD's PF queues](https://bsdly.blogspot.ca/2017/04/forcing-password-gropers-through.html), [Badness, Enumerated by Robots](https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html) (blog posts) + [The Book of PF](https://www.nostarch.com/pf3) --- ## Scenario: Wordpress site - You can re-use the *bruteforce* table or make a separate one, like ```shell table <web_brutes> persist counters block from <web_brutes> ``` - Now change the **$webports** rule to ```shell * pass proto tcp to port $webports flags S/SA keep state \ * (max-src-conn 15, max-src-conn-rate 5/10, overload <web_brutes> flush global) ``` - Alternatively, on a machine (the gateway) that does not run Wordpress, do ```shell grep wp-login /var/www/logs/access.log | awk '{print $1}' | \ sort -u | xargs doas pfctl -t web_brutes -T add ``` - (**Wordpressers**: Do these look right? Consult web logs and [tcpdump](https://man.openbsd.org/tcpdump) output.) ??? max-src-conn - see man page but max number of simultaneous connections from one host max-src-conn-rate - number of new connections per number of seconds You could also look into max-src-nodes, max-src-states --- ## Annoying spammers with spamd - [spamd(8)](https://man.openbsd.org/spamd) is good, clean, fun <br><br> - Speaks enough SMTP to do [greylisting](https://en.wikipedia.org/wiki/Greylisting) <br><br> - Can tarpit known bad senders and generate blacklists by greytrapping <br><br> - Default [spamd.conf](https://man.openbsd.org/spamd.conf) gives you one blacklist import and basics - (**Hint:** no real SMTP service required) <br><br> - You can even generate your own blacklists by *greytrapping* via *non-deliverable* spamtrap addresses in your own domain(s) <br><br> - Also see: - [The Book of PF](https://www.nostarch.com/pf3) - [In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes](https://bsdly.blogspot.ca/2012/05/in-name-of-sane-email-setting-up-spamd.html) - [Maintaining A Publicly Available Blacklist](https://bsdly.blogspot.ca/2013/04/maintaining-publicly-available.html) --- ## Annoying spammers with spamd - Set up tables - Divert traffic to the spamd process - Only let the "good guys" pass to the real SMTP daemon ```shell table <spamd-white> persist table <nospamd> persist file "/etc/mail/nospamd" pass in on egress proto tcp to any port smtp divert-to 127.0.0.1 port spamd pass in on egress proto tcp from <nospamd> to any port smtp pass in log on egress proto tcp from <spamd-white> to any port smtp pass out log on egress proto tcp to any port smtp ``` - and then run ```shell $ doas rcctl enable spamd $ doas rcctl start spamd ``` **TIP:** check out [smtpctl spf walk <_nospamd_domains.txt_](https://man.openbsd.org/smtpctl) to feed your _nospamd_ table from live [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) data (in OpenBSD 6.3 onwards), see the blog post [Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting](https://bsdly.blogspot.com/2018/11/goodness-enumerated-by-robots-or.html) --- ## Scenario: SYN flood vs syncookies A common Denial-of-Service (DOS) technique is to send large numbers of SYNs from spoofed addresses, filling up the state table. In OpenBSD 6.3 and newer we have the [pf.conf](https://man.openbsd.org/pf.conf) option ```shell set syncookies ``` The default is off, if you enable with ```shell set syncookies on ``` **all** SYNs get SYNACK answer, but no resources allocated until ACK received (pending match to **pass** rule) The other option is **adaptive** with syncookies used only when half open TCP connections reach the **start** percentage, until the **end** level is reached ```shell set syncookies adaptive (start 29%, end 15%) ``` --- ## Consider: what about that separate DMZ? - What do you need? <br><br><br> - IP addresses: Segment off or allocate separate address ranges <br><br><br> - Attach each segment to separate interface, VLAN <br><br><br> - Do the ruleset surgery - which services do you run, where - do you need renumbering? - what traffic (in and out) is actually required? ??? --- class: center, middle # Questions ? ??? Let's ask if there are any questions before continuing. Make sure we have everyone onboard. --- class: center, middle # Exercise 3 ### Offering services --- ## Excercise 3 - Goals - You're now offering services <br><br><br> - **Host 1** will provide **http** service <br><br><br> - **Host 2** will provide **smtp** service <br><br><br> - We need to setup: - The services - Redirects - Firewall rules --- background-image: url(images/exercise3.png) class: right ## Exercise 3 - Network ??? This is not exactly the network we have, but could be thought as such. Host1 and Host2 are in what could be considered our DMZ. --- ## Exercise 3 - on Host1 - We need to configure and start httpd ```shell # cp /etc/examples/httpd.conf /etc/httpd.conf < comment out the HTTPS part > # rcctl enable httpd # rcctl start httpd httpd(ok) ``` --- ## Exercise 3 - on Host2 Stop the *smtpd* daemon ```shell # rcctl stop smtpd ``` - Change the config to listen on all interfaces: - Change the appropriate line in ### /etc/mail/smtpd.conf ```shell listen on all ``` <br><br> - Then start the daemon ```shell # rcctl enable smtpd # rcctl start smtpd smtpd(ok) ``` - (It might take a while) --- ## Exercise 3 - on gateway ### /etc/pf.conf ```shell webserver_v4 = "$IP_addr_of_host1" webserver_v6 = "fd18:b5d:XX::80" webports = "{ http, https }" emailserver_v4 = "$IP_addr_of_host2" emailserver_v6 = "fd18:b5d:XX::25" email = "{ smtp, pop3, imap, imaps, pop3s }" match in on egress inet proto tcp to egress port $webports rdr-to $webserver_v4 match in on egress inet proto tcp to egress port $email rdr-to $emailserver_v4 pass inet proto tcp to $webserver_v4 port $webports pass inet proto tcp to $emailserver_v4 port $email pass log inet proto tcp from $emailserver_v4 to port smtp pass inet6 proto tcp to $webserver_v6 port $webports pass inet6 proto tcp to $emailserver_v6 port $email pass log inet6 proto tcp from $emailserver_v6 to port smtp ``` - **NB:** No redirects are needed for IPv6 --- ## Exercise 3 - checks - Try connecting to the HTTP and SMTP port of your friends/neighbours: <br><br> - From Gateway: ```shell telnet -6 fd18:b5d:XX::80 80 telnet -4 10.255.255.XX 80 ``` <br><br> - and ```shell telnet -6 fd18:b5d:XX::25 25 telnet -4 10.255.255.XX 25 ``` --- ## Tips - Decide your network topology - DMZ (?) - Multi-customer (?) - Multi-customer, Multi-DMZ(?) - Segment off your subnets - IPv4 (Do you NAT)? - IPv6 - Do you do NAT64? - Per subnet (customer) - Which services do you expose? - Write the rules - pamper^H^H^H^H^Hproxying --- class: center, middle # Traffic Shaping ### Section 4 --- ## Traffic shaping - OpenBSD has three separate shaping techniques: - *priorities* (set prio), introduced in OpenBSD 5.0 - *queues*, introduced in OpenBSD 5.5, and - *flows*, introduced in OpenBSD 6.2 (aka FQ-CoDel) <br><br> - **Remember:** - Traffic shaping is about *dropping packets* - But is only relevant when there's a *reason* to start dropping - Then you get to pick which ones using the traffic shaping tools <br><br> - Works only one way, **outbound** --- ## Traffic shaping with priorities - In OpenBSD, every packet has a priority <br><br> - Possible values are 0 (garbage) through 7 (want!) - Default for most traffic is 3. <br><br> - So if you want *all* ssh traffic to move ahead of other traffic you could do a <br><br> ```shell pass proto tcp to port ssh set prio 6 ``` <br><br> - and assign specific, non-default ( != 3 ) values to others. --- ## Beating the FIFO with prio - By default, packets are serviced on a first come, first served basis - or 'First in, First out' (FIFO). <br> - But: TCP wants ACKs for sent packets within a reasonable time - Otherwise the packet is considered lost, and retransmit will follow (transfer stalls). <br> - ACKs are tiny and have their TOS set to lowdelay, and this trick will cheat the FIFO: <br> ```shell match out on $ext_if proto tcp from $ext_if set prio (3, 7) match in on $ext_if proto tcp to $ext_if set prio (3, 7) ``` <br> as per the [man page](https://man.openbsd.org/pf.conf), <br><br> *If two priorities are given, TCP ACKs with no data payload and packets which have a TOS of lowdelay will be assigned to the second one. Packets with a higher priority number are processed first, and packets with the same priority are processed in the order in which they are received.* <br> - See [Prioritizing empty TCP ACKs with pf and ALTQ](https://www.benzedrine.ch/ackpri.html) by Daniel Hartmeier for the ALTQ way --- ## FQ-CoDel flows for fair bandwidth sharing Introduced in [OpenBSD 6.2](https://www.openbsd.org/62.html), the FQ-CoDel algorithm (see [RFC8290](https://tools.ietf.org/html/rfc8290)) defines **flows** to set up fair sharing for a specified number of simultaneous connections Enable for your configuration with something like ```shell queue outq on bge0 bandwidth 18M max 18M flows 1024 qlimit 1024 \ default ``` Estimate your approximate high number of simultaneously actively transmitting sessions, put that number in (up to 32767) --- ## Shaping with HFSC queues - fixed sizes - When priorities don't quite cut it, you can slice your bandwidth into queues. <br><br> - For static shaping, give bandwidth values in absolute values: <br><br> - Only leaf queues can be assigned traffic - make sure allocations sum up to parent queue allocation <br><br> - Unless quotas approach saturation, no actual shaping (dropping) will take place --- ## Shaping with HFSC queues - fixed sizes <br><br><br> ```shell queue main on $ext_if bandwidth 20M queue defq parent main bandwidth 3600K default queue ftp parent main bandwidth 2000K queue udp parent main bandwidth 6000K queue web parent main bandwidth 4000K queue ssh parent main bandwidth 4000K queue ssh_interactive parent ssh bandwidth 800K queue ssh_bulk parent ssh bandwidth 3200K queue icmp parent main bandwidth 400K ``` --- ## Shaping with HFSC queues - fixed assignment - You can either do queue assignment with match rules: ```shell match log quick on $ext_if proto tcp to port ssh \ queue (ssh_bulk, ssh_interactive) match in quick on $ext_if proto tcp to port ftp queue ftp match in quick on $ext_if proto tcp to port www queue http match out on $ext_if proto udp queue udp match out on $ext_if proto icmp queue icmp ``` - treat filtering (block, pass) in separate rules, - *or* - append 'set queue' to individual pass rules - Any traffic not explicitly assigned goes in the default queue --- ## Shaping with HFSC queues - flexible allocations (min, max, burst) You can build in flexibility: ```shell queue rootq on $ext_if bandwidth 20M queue main parent rootq bandwidth 20479K min 1M max 20479K qlimit 100 queue qdef parent main bandwidth 9600K min 6000K max 18M default queue qweb parent main bandwidth 9600K min 6000K max 18M queue qpri parent main bandwidth 700K min 100K max 1200K queue qdns parent main bandwidth 200K min 12K burst 600K for 3000ms queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300 ``` Note here the added *min* and *max* values: combinded queue bandwidth can exceed actual sum; this gets you upper and lower bound within physical limits. *burst N for M* - allow bursts of that size and length *qlimit* - size of the queue's holding buffer - larger values *may* delay (packets are kept longer before sending) --- ## Systat - Available on all the BSDs - OpenBSD version has added functionalities - Offers view into your system's traffic - Live queues, states, rules - Check how your rules are behaving on your system in realtime - More info in the man page [systat(1)](https://man.openbsd.org/systat) --- ## queue monitoring - systat queues <br><br><br> ```shell 1 users Load 2.56 2.27 2.28 skapet.bsdly.net 20:55:50 QUEUE BW SCH PRI PKTS BYTES DROP_P DROP_B QLEN BOR SUS P/S B/S rootq on bge0 20M 0 0 0 0 0 0 0 main 20M 0 0 0 0 0 0 0 qdef 9M 6416363 2338M 136 15371 0 462 30733 qweb 9M 431590 144565K 0 0 0 0.6 480 qpri 2M 2854556 181684K 5 390 0 79 5243 qdns 100K 802874 68379K 0 0 0 0.6 52 spamd 1K 596022 36021K 1177533 72871514 299 2 136 ``` or --- ## queue monitoring - pfctl ```shell $ doas pfctl -vsq [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 ] queue rootq on bge0 bandwidth 20M qlimit 50 [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 ] queue main parent rootq bandwidth 20M, min 1M, max 20M qlimit 100 [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/100 ] queue qdef parent main bandwidth 9M, min 8M, max 18M default qlimit 50 [ pkts: 6517150 bytes: 2458545319 dropped pkts: 136 bytes: 15371 ] [ qlength: 0/ 50 ] queue qweb parent main bandwidth 9M, min 8M, max 18M qlimit 50 [ pkts: 431741 bytes: 148072219 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 ] queue qpri parent main bandwidth 2M, min 700K, max 2M burst 4M for 3000ms qlimit 50 [ pkts: 2855418 bytes: 186101241 dropped pkts: 5 bytes: 390 ] [ qlength: 0/ 50 ] queue qdns parent main bandwidth 100K, min 12K burst 600K for 3000ms qlimit 50 [ pkts: 803548 bytes: 70079760 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 ] queue spamd parent rootq bandwidth 1K, max 1K qlimit 300 [ pkts: 596424 bytes: 36910940 dropped pkts: 1178456 bytes: 72928604 ] [ qlength: 300/300 ] ``` add another v ([pfctl](https://man.openbsd.org/pfctl) -vvsq) for continuously updating display --- class: center, middle # Questions ? ??? Let's ask if there are any questions before continuing. Make sure we have everyone onboard. --- class: center, middle # Exercise 4 ### Queueing --- ## Exercise 4 - Goals - With the configs from exercise 3, now add: <br><br><br> - A set of queues, and <br><br><br> - Statements to add rules to the queues --- ## Exercise 4 - on Gateway - Configure the queues <br><br> ### /etc/pf.conf ```shell queue rootq on $ext_if bandwidth 20M queue main parent rootq bandwidth 20479K min 1M max 20479K qlimit 100 queue defq parent main bandwidth 9600K min 6000K max 18M default queue http parent main bandwidth 9600K min 6000K max 18M queue smtp parent main bandwidth 9600K min 6000K max 18M queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300 ``` --- ## Exercise 4 - on Gateway - and then apply them to the match statements ### /etc/pf.conf ```shell match in on egress inet proto tcp to egress port $webports rdr-to $webserver_v4 \ queue http match in on egress inet proto tcp to egress port $email rdr-to $emailserver_v4 \ queue smtp pass inet6 proto tcp to $webserver_v6 port $webports set queue http pass inet6 proto tcp to $emailserver_v6 port $email set queue smtp pass log inet6 proto tcp from $emailserver_v6 to port smtp set queue smtp ``` --- ## Exercise 4 - Check - Check the queues have been effectively created ```shell # systat queues ``` <br><br><br> - or, alternatively ```shell # pfctl -vsq ``` --- class: center, middle # Tips ### Section 5 --- # Choosing your ISP, a quick guide - Are they national or regional IX members? <br><br> - Do they have geographical redundancy ? - or do you need to arrange that for yourself ? <br><br> - Do they actually understand your questions about peering, routing, multiple paths? - (avoid consumer oriented SOHO-only shops) <br><br> - Do they _suck_? --- ## Getting transit - Find well peered transit providers - Can improve quality and shorten AS paths - No capacity problems <br><br><br> - Find your top traffic destinations: - Can improve quality - Peer with them or find closer upstream - Traffic profile from flow collectors can be useful --- ## Common mistakes - No diversity - All reached over same cable - All connect to the same transit - All have poor onward transit and peering arrangements <br><br><br> - Signing up with too many transit providers - Lots of small circuits - These cost more per Mbps than larger ones --- ## Basic OpenBGPd configuration, operation and interaction with PF - **B**order **G**ateway **P**rotocol - Manage and exchange route information with BGP peers - Once you have the ASn registered, do the basic config. <!-- - You will be in [phessler's session](https://www.bsdcan.org/2019/schedule/events/1087.en.html), right? --> - In your *pf.conf*: - enable BGP to pass between your routers and your peers' -- **TCP and UDP 179** - **Neat trick**: Define tables in your [pf.conf](https://man.openbsd.org/pf.conf) - bgpd maintains them via **pftable** attributes on [bgpd.conf](https://man.openbsd.org/bgpd.conf) objects --- ## Use cases for OSPF, BGP or ECMP - **OSPF:** **O**pen **S**hortest **P**ath **F**irst - is a IGP **I**nterior **G**ateway **P**rotocol - Each router maintains link state information for links and networks within your AS - Calculates routing cost - Use [ospf6d](https://man.openbsd.org/ospf6d) for IPv6 - Use [ospfd](https://man.openbsd.org/ospfd) for IPv4 - Need to *pass proto ospf* between routers. - **BGP:** announces and receives routes - can be both an IGP or EGP **E**xterior **G**ateway **P**rotocol - highly scalable (Internet scale) - can be used for signaling and sending additional information with route announcements - Use [bgpd](https://man.openbsd.org/bgpd) - need to *pass proto tcp port 179* between routers --- ## Use cases for OSPF, BGP or ECMP (cont) - **ECMP:** **E**qual **C**ost **M**ulti-**P**ath - target reachable via more than one route - load distribution or redundancy over multiple links - **Tip** Use [ifstated](https://man.openbsd.org/ifstated) to handle link downtime. --- ## BCP38, MANRS and Internet peering "[**BCP38**](https://tools.ietf.org/html/bcp38)" -- Discussed also in another effort **M**utually **A**greed **N**orms for **R**outing **S**ecurity (MANRS) - Define four concrete actions network operators should implement - Coordination - Keep your contacts updated - Validation - Route objects, RPKI, BGPSec - Anti-spoofing - uRPF - Filtering on external Interfaces facing external suppliers - Drop inbound Traffic with a src IP claiming to be from your networks / private networks. - Drop outbound Traffic with a src IP address that is not in your Public IP network range. - Build a visible community of security-minded operators - Valuable resource: [The Routing Manifesto](https://www.routingmanifesto.org/) --- ## Introducing VXLAN in your network [vxlan](https://man.openbsd.org/vxlan) - the **V**irtual e**X**tensible **L**ocal **A**rea **N**etwork tunnel interface <br><br> - Pushes layer 2 network (Ethernet frames) over layer 3 (IP) tunnels - 24-bit *vnetid* (vs max 4k VLANs) <br><br> - Has *no* built in security - Intended for '*trusted*' (Datacenter, inter-hypervisor) environments - Otherwise, consider transport over IPSEC. <br><br> - Default transport over **UDP 4789** (aka **vxlan**) - make sure that traffic passes between endpoints --- # Introducing VXLAN in your network ```shell # ifconfig vxlan0 tunnel 192.168.100.101 192.168.200.201 vnetid 17 # ifconfig vxlan0 10.11.12.100/24 ``` <br><br> ```shell # ifconfig vxlan0 tunnel 192.168.200.201 192.168.100.101 vnetid 17 # ifconfig vxlan0 10.11.12.101/24 ``` <br><br> ```shell table <vxendpoints> { 192.168.200.201 192.168.200.204 } pass from <vxendpoints> to port vxlan ``` <br><br> Buy [Reyk](https://www.bsdcan.org/2017/schedule/speakers/227.en.html) a beer. --- ## Readable and maintainable toolsets - **Macros** - descriptive names, keep uniform - **Tables** - descriptive names - consider daemon/scripting interface - **Interface groups** - you know egress already - make your own and filter on them - **Anchors** - group rules by common criteria - tagging - interface or group - Service names vs port numbers - **Comments** - yes, you **will** forget why this was a good idea --- ## Useful 3rd party packages (ports) for OpenBSD OpenBSD base operating system can be supplimented by the following packages and features: - pftop - a curses-based utility for real-time display of active states and rules for pf. It is a cross between top and pfctl -sr and pfctl -ss. - pftop can be installed with the following command <code>pkg_add pftop</code> - nsh **n**etwork **sh**ell - nsh can be installed with the following command <code>pkg_add nsh</code> --- ## Now let's add wireless - Wireless used to be hard, (WPA in particular), now it's 'just another interface' <br><br> - 802.11* support in OpenBSD has a,b,g,n, ac only in some drivers ([bwfm(4)](https://man.openbsd.org/bwfm), [iwx(4)](https://man.openbsd.org/iwx)) <br><br> - Not all drivers support hostap - check man pages before buying kit for access point use <br><br> - Optionally setup with commercial APs for radio part - do DHCP, filtering, authentication and so forth from OpenBSD --- class: center, middle # Questions ? ??? Let's ask if there are any questions before continuing. Make sure we have everyone onboard. --- class: center, middle # Troubleshooting ### Section 6 #### "It's all your fault. Until you track down and fix the root cause." --- ## Troubleshooting 101: ICMP(v6) - ICMP: **I**nternet **C**ontrol **M**essage **P**rotocol - The *ping of death* scare is almost over, let's enable [ping](https://man.openbsd.org/ping): <br><br> ```shell icmp_types = "{ echoreq, unreach }" pass inet proto icmp all icmp-type $icmp_types keep state pass inet proto icmp from $localnet icmp-type $icmp_types pass inet proto icmp to $ext_if icmp-type $icmp_types pass inet6 proto icmp6 from $localnet icmp6-type $icmp6_types pass inet6 proto icmp6 to $ext_if icmp6-type $icmp6_types ``` <br><br> - **echoreq**: lets [ping](https://man.openbsd.org/ping) do its thing - **unreach**: lets you do _path MTU discovery_ (PMTUD) --- ## Troubleshooting 101: ICMP not all messages created equal! - Some ICMP messages are frivolous: - Type 4 — Source Quench (Deprecated) - Type 5 — Redirect (if you need this you are doing it wrong) - Type 6 — Alternate Host Address (Deprecated) - Type 13 — Timestamp - Type 14 — Timestamp Reply - Type 15 — Information Request (Deprecated) - Type 16 — Information Reply (Deprecated) - Type 17 — Address Mask Request (Deprecated) - Type 18 — Address Mask Reply (Deprecated) --- ## Troubleshooting 101: basic diagnostics built in in OpenBSD base Basic IP connectivity ```shell ping <host> ``` ```shell traceroute <host> ``` ```shell telnet <host> <tcp port> ``` ```shell tcpdump -i <interfacename> ``` - netcat command - nc ```shell nc -z <host> <startport>-<endport> ``` --- ## Troubleshooting 101: not all pings were created equal! ping packets can be manipulated in many ways - set a specific source address (useful for routers) ```shell ping [-I <src>address> ] <destaddress> ``` -set ping packet size (MTU path Discovery Issues) ```shell ping [-s <packet-size>] <destaddress> ``` - set dont fragment bit (MTU / Fragmentation issues) ```shell ping [ -D ] <destaddress> ``` -set ping packet size (MTU / fragmentation issues ```shell ping [-s <packet-size>] <destaddress> ``` --- ## Troubleshooting 101: basic diagnostics built in in OpenBSD base DNS functionality basic commands ```shell ping <hostdnsname> ``` ```shell telnet <hostdnsname> <tcp port> ``` - in depth dns testing ```shell dig <hostdnsname> ``` ```shell nslookup <hostdnsname> ``` --- ## Troubleshooting 101: diagnostic tools available in ports IP connectivity - mtr (not the gtk version) ```shell mtr <host> ``` - tcp traceroute ```shell tcptraceroute <host> ``` --- ## Troubleshooting 101: Statistics - Statistics can be had with **[pfctl](https://man.openbsd.org/pfctl) -s info** For statistics (bytes/packets passed per rule) attach _labels_ per rule ```shell pass log proto { tcp, udp } to $emailserver port smtp label "mail-in" pass log proto { tcp, udp } from $emailserver to port smtp label "mail-out" ``` ```shell $ doas pfctl -vs rules pass inet proto tcp from any to 192.0.2.225 port = smtp flags S/SA keep state label "mail-in" [ Evaluations: 1664158 Packets: 1601986 Bytes: 763762591 States: 0 ] [ Inserted: uid 0 pid 24490 ] pass inet proto tcp from 192.0.2.225 to any port = smtp flags S/SA keep state label "mail-out" [ Evaluations: 2814933 Packets: 2711211 Bytes: 492510664 States: 0 ] [ Inserted: uid 0 pid 24490 ] ``` --- ## Troubleshooting 101: Statistics - If you need to pass the data to a script - Or a database - A graphing engine <br><br><br> ```shell $ doas pfctl -zvsl mail-in 1664158 1601986 763762591 887895 682427415 714091 81335176 mail-out 2814933 2711211 492510664 1407278 239776267 1303933 252734397 ``` --- ## Troubleshooting 101: log to pflog Rules with the **log** keyword log packet data to the [pflog](https://man.openbsd.org/pflog) device(s) ```shell # log blocked packets block log(all) # logs initial packet of matching connections: pass log proto tcp to port ssh # logs all matching packets: pass log(all) proto tcp to port ssh log(all) # logs matches on this and all succeeding rules pass log(matches) proto tcp to port ssh # logs all packets matches on this and all succeeding rules pass log(all, matches) proto tcp to port ssh ``` ```shell match log(all, matches) # log *everything* ``` --- ## Troubleshooting 101: tcpdump, read from pflog - [tcpdump](https://man.openbsd.org/tcpdump) is your friend - Let it loose on the pflog device: ```shell $ doas tcpdump -n -e -ttt -i pflog0 tcpdump: WARNING: snaplen raised from 116 to 160 tcpdump: listening on pflog0, link-type PFLOG May 29 21:06:27.165561 rule def/(match) pass in on bge1: 192.168.103.126.15526 > 213.187.179.198.22: . ack 2951513182 win 16332 (DF) [tos 0x10] May 29 21:06:27.166934 rule 16/(match) pass in on bge0: 158.36.191.135.22 > 213.187.179.198.59516: . ack 1734404306 win 64800 [tos 0x8] May 29 21:06:27.166939 rule 2/(match) match in on bge0: 158.36.191.135.22 > 213.187.179.198.59516: . ack 1 win 64800 [tos 0x8] May 29 21:06:27.168340 rule def/(match) pass out on bge1: 213.187.179.198.22 > 192.168.103.126.15526: P 69:153(84) ack 0 win 17520 [tos 0x10] May 29 21:06:27.169150 rule def/(match) pass out on bge1: 213.187.179.198.22 > 192.168.103.126.15526: P 153:333(180) ack 0 win 17520 [tos 0x10] May 29 21:06:27.169265 rule def/(match) pass out on bge1: 213.187.179.198.22 > ``` - **NB** rule number, matches your *loaded* rule set --- ## Troubleshooting 101: Hitting and avoiding limits - On busy systems, you may need to raise limits from default values - Check with: ```shell $ doas pfctl -s info ``` - versus the output of **pfctl -s memory** and **pfctl -s timeouts** - You may need to bump up from defaults: ```shell # increase state limit from 10'000 states on busy systems set limit states 100000 # increase no of source nodes set limit src-nodes 100000 ``` --- ## Troubleshooting 101: quick flow analysis using iftop (package) - Sometimes we need to check what traffic is flowing on a given interface - iftop ncurses package can be installed and allows visibility on top traffic flows on an interface - iftop allows that quick diagnostics check without needing a full flow anlaysis infrastructure - iftop ncurses package can be installed on any OpenBSD system using the following command ```shell pkg_add iftop ``` --- ## Troubleshooting 101: using iftop (package) example - Checking flow on the first intel 1Gbs interface on a router ```shell iftop -i em0 ``` - command output ```shell 1.91Mb 3.81Mb 5.72Mb 7.63Mb 9.54Mb └───────────────┴───────────────┴───────────────┴───────────────┴─────────────── 10.0.2.15 => di-in-f190.1e100.net 23.5Kb 11.8Kb 2.95Kb <= 3.22Mb 805Kb 201Kb 10.0.2.15 => dj-in-f106.1e100.net 0b 18.1Kb 6.53Kb <= 0b 708Kb 218Kb 10.0.2.15 => dj-in-f94.1e100.net 0b 4.41Kb 1.10Kb <= 0b 57.5Kb 14.4Kb 10.0.2.15 => dj-in-f100.1e100.net 0b 4.14Kb 1.03Kb <= 0b 43.3Kb 10.8Kb 10.0.2.15 => dh-in-f95.1e100.net 5.69Kb 4.36Kb 1.60Kb <= 133Kb 29.5Kb 8.06Kb 10.0.2.15 => dj-in-f119.1e100.net 0b 3.26Kb 834b <= 0b 19.6Kb 4.91Kb 10.0.2.15 => ig-in-f94.1e100.net 0b 4.66Kb 1.16Kb <= 0b 12.8Kb 3.21Kb 10.0.2.15 => dj-in-f148.1e100.net 12.5Kb 2.49Kb 638b <= 49.9Kb 9.99Kb 2.50Kb 10.0.2.15 => dg-in-f102.1e100.net 0b 6.57Kb 1.64Kb <= 0b 5.51Kb 1.38Kb ──────────────────────────────────────────────────────────────────────────────── TX: cum: 832KB peak: 89.6Kb rates: 51.7Kb 62.4Kb 20.1Kb RX: 20.6MB 3.43Mb 3.43Mb 1.66Mb 472Kb TOTAL: 21.4MB 3.48Mb 3.48Mb 1.72Mb 492Kb ``` --- ## Troubleshooting 101: netflow aka pflow (IPFIX) - Records TCP/IP *flow* metadata - srcIP - dstIP - (srcPort, dstPort) - startTime - endTime - Packets - Bytes - OpenBSD has the [pflow(4)](https://man.openbsd.org/pflow) virtual network interface - which generates the datagrams from the state table - Useful for network monitoring, DDoS protection, etc. --- ## Troubleshooting 101: netflow setup - Set up a *sensor*: ```shell $ cat /etc/hostname.pflow0 flowsrc 192.168.103.1 flowdst 192.168.103.252:9995 pflowproto 10 ``` - Then configure your *collector* at the **flowdst** IP address for analysis and network overlordship. <br><br> - Lots of collector options available in ports: nfsen, flow-tools, pmacct, FastNetMon and others. - More info: - Michael W. Lucas: [Network Flow Analysis](https://www.nostarch.com/networkflow) - and Peter N. M. Hansteen: [Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen](https://bsdly.blogspot.ca/2014/02/yes-you-too-can-be-evil-network.html). --- ## Flow Analyser example Fastnetmon - Example of a typcial flow anlayser software fastnetmon: - User can view FastNetMon statistics via the CLI client fastnetmon_client ```shell # fastnetmon_client FastNetMon 1.1.7 master git- Try Advanced edition: https://fastnetmon.com IPs ordered by: packets Incoming traffic 1505664 pps 15397 mbps 85 flows 37.203.[redacted] 59184 pps 485 mbps 0 flows 37.203.[redacted] 45040 pps 504 mbps 0 flows 37.203.[redacted] 26924 pps 270 mbps 0 flows 185.55.[redacted] 24211 pps 240 mbps 0 flows 5.134.[redacted] 23872 pps 290 mbps 0 flows 45.11.[redacted] 23634 pps 250 mbps 0 flows 185.55.[redacted] 22451 pps 255 mbps 0 flows 45.11.[redacted] 20943 pps 254 mbps 0 flows 185.55.[redacted] 20298 pps 246 mbps 0 flows 5.134.[redacted] 20188 pps 236 mbps 0 flows ``` - With FastNetMon one can implement mitigations based on tresholds - Packets per second pps - Bandwidth per second Mbps --- class: center, middle # Exercise 5 ### NAT64 --- ## NAT64 - Single-stack clients will only have IPv6 <br><br> - Translator box will strip all headers and replace them with IPv4 <br><br> - Requires some DNS “magic” - Capture responses and replace A with AAAA - Response is crafted based on target IPv4 address <br><br> - Usually implies address sharing on IPv4 --- background-image: url(images/nat64.png) --- ## 464-XLAT - Extension to NAT64 to access IPv4-only applications (like Skype or Whatsapp) <br><br><br> - Handset pretends there is an IPv4 address (CLAT) and sends IPv4 packets in UDP over IPv6 --- background-image: url(images/464-xlat.png) --- ## Exercise 5 - Goals - Define the translating prefix - 64:ff9b::/96 is reserved by IETF - Add NAT64 configuration to PF - Remove IPv4 from the internal network - Configure DNS64 on unbound --- ## Exercise 5 - _pf.conf_ ```shell pass in quick on $int_if inet6 from any to 64:ff9b::/96 \ af-to inet from (egress:0) keep state ``` - _unbound.conf_ ```shell module-config: "dns64 validator iterator" dns64-prefix: 64:FF9B::/96 ``` and it's done! --- class: center, middle # Questions ? ### Last chance... ### or questions@pftutorial.net ??? Let's ask if there are any questions before continuing. Make sure we have everyone onboard. --- ## Web accessible resources ### OpenBSD website and documentation [https://www.openbsd.org/](https://www.openbsd.org/) The official OpenBSD website – to donate: [https://www.openbsd.org/donations.html](https://www.openbsd.org/donations.html) and please do donate, corporates may prefer [https://www.openbsdfoundation.org/](https://www.openbsdfoundation.org/) - a Canadian non-profit [The PF User Guide on the OpenBSD web site](https://www.openbsd.org/faq/index.html) [OpenBSD online man pages](https://man.openbsd.org/) Note: You can convert the man page of pf.conf to PDF for reading in your favourite reader with the command: <code>man -T pdf pf.conf > pf.conf.pdf </code> --- ## Resources ### Books / e-Books Michael W Lucas: [Absolute OpenBSD, 2nd ed.](https://www.nostarch.com/openbsd2e) Peter N. M. Hansteen: [The Book of PF, 3rd ed.](https://www.nostarch.com/pf3) Elizabeth D. Zwicky, Simon Cooper, D. Brent Chapman [Building Internet Firewalls, 2nd ed.](https://www.oreilly.com/library/view/building-internet-firewalls/) ### Blogs [https://undeadly.org/](https://undeadly.org/) - The OpenBSD Journal news site [https://bsdly.blogspot.com/](https://bsdly.blogspot.com/) - Peter's rants^H^H^H^H^Hblog posts [https://www.tedunangst.com/flak/](https://www.tedunangst.com/flak/) tedu@ on developments --- background-image: url(images/end.png) - ??? Notes for this slide --- background-image: url(images/end2.png) - ??? Notes for this other slide ---