{"id":2707,"date":"2020-08-21T09:26:19","date_gmt":"2020-08-21T08:26:19","guid":{"rendered":"https:\/\/beufa.net\/?p=2707"},"modified":"2020-08-21T09:55:17","modified_gmt":"2020-08-21T08:55:17","slug":"export-netflow-from-unifi-gateway","status":"publish","type":"post","link":"https:\/\/beufa.net\/fr\/blog\/export-netflow-from-unifi-gateway\/","title":{"rendered":"Exporter les netflows depuis une gateway Unifi"},"content":{"rendered":"\n

Context<\/h2>\n\n\n\n

Looking on which client in my home network is consuming bandwith and trying to catch \/ debug some attacks and network flows, I played with Unifi gateway (Model UniFi Security Gateway 3P) to export netflow data to a pmacct\/nfacctd netflow receiver.<\/p>\n\n\n\n

Configuration<\/h2>\n\n\n\n

First you need to define netflow \/ flow-accounting configuration on Unifi gateway.<\/p>\n\n\n\n

CLI<\/h3>\n\n\n\n

For testing purpose, you could enable temporary the netflow export using a manual configuration using direct configuration on CLI. Take care, on each change on your unifi controller, configuration manually set on the gateway will be erased.<\/p>\n\n\n\n

user@gw:~$ configure\n[edit]\nuser@gw# show system flow-accounting \n ingress-capture pre-dnat\n interface eth0\n interface eth1\n netflow {\n     sampling-rate 10\n     server 192.168.1.100 {\n         port 6999\n     }\n     timeout {\n         expiry-interval 60\n         flow-generic 60\n         icmp 300\n         max-active-life 604800\n         tcp-fin 300\n         tcp-generic 3600\n         tcp-rst 120\n         udp 300\n     }\n     version 9\n }\n syslog-facility daemon\n[edit]\n<\/pre><\/div>\n\n\n\n
Unifi controller<\/h3>\n\n\n\n
In order to deploy this configuration automatically using the Unifi controller, and to be kept after a reboot or any configuration change, you should create a config.gateway.json <\/em>file to automate \/ keep this deployment on each provisioning, on each site ID folder :<\/p>\n\n\n\n
<\/path><\/svg><\/div>
\u2718 ⚡ root@controller \ue0b0 \/var\/lib\/unifi\/sites\/mysiteID \ue0b0 cat config.gateway.json\n{\n        \"system\": {\n                \"flow-accounting\": {\n                        \"ingress-capture\": \"pre-dnat\",\n                        \"interface\": [\n                                \"eth0\",\n                                \"eth1\"\n                        ],\n                        \"netflow\": {\n                                \"sampling-rate\": \"10\",\n                                \"server\": {\n                                        \"192.168.1.100\": {\n                                                \"port\": \"6999\"\n                                        }\n                                },\n                                \"timeout\": {\n                                        \"expiry-interval\": \"60\",\n                                        \"flow-generic\": \"60\",\n                                        \"icmp\": \"300\",\n                                        \"max-active-life\": \"604800\",\n                                        \"tcp-fin\": \"300\",\n                                        \"tcp-generic\": \"3600\",\n                                        \"tcp-rst\": \"120\",\n                                        \"udp\": \"300\"\n                                },\n                                \"version\": \"9\"\n                        },\n                        \"syslog-facility\": \"daemon\"\n                }\n        }\n}\n<\/pre><\/div>\n\n\n\n
With this configuration deployed, you should be able to receive UDP netflow export on port 6999, with a 1 for 10 sample rate. It means 1 out 10 packets will be sampled to the netflow exporter \/ receiver. This value should be adjusted carefully to avoid CPU exhaustion on your Unifi gateway (processor is some old stuff, with no powerful capacities)<\/p>\n\n\n\n
Data Collection<\/h2>\n\n\n\n
I’ve always been a pmacct<\/a>‘s project fan \ud83d\ude09 So naturally, I deployed a nfacctd<\/em> daemon on my home server to collect netflow<\/em> exports and aggregate them using some primitives<\/a>. <\/p>\n\n\n\n
<\/path><\/svg><\/div>
user@home \ue0b0 ~ \ue0b0 cat \/opt\/pmacct\/etc\/nfacctd.conf       \n! nfacctd configuration for home\n!\n! nfacctd daemon parameters\n! for details : http:\/\/wiki.pmacct.net\/OfficialConfigKeys\n!\ndaemonize: false\npidfile: \/var\/run\/nfacctd.pid\nlogfile: \/var\/log\/nfacctd.log\nsyslog: daemon\ncore_proc_name: nfacctd\n!\n\n!interface: enp0s25\nnfacctd_ip: 192.168.1.100\nnfacctd_port: 6999\n!nfacctd_time_new: true\n!nfacctd_disable_checks: true\n!nfacctd_renormalize: true\nnfacctd_ext_sampling_rate: 10\n\ngeoip_ipv4_file: \/usr\/share\/GeoIP\/GeoIP.dat\ngeoip_ipv6_file: \/usr\/share\/GeoIP\/GeoIPv6.dat\n\n!aggregate[traffic]: in_iface,out_iface,proto,tcpflags,ethtype,src_host_country,dst_host_country,src_host,dst_host\naggregate[traffic]: in_iface,out_iface,proto,tcpflags,src_host_country,dst_host_country,src_host,dst_host\n\n\n! storage methods\nplugins: print[traffic]\n\n! print rules\nprint_output_file[traffic]: \/tmp\/nfacctd\/%s.json\nprint_output: json\nprint_history: 2m\nprint_history_roundoff: m\nprint_refresh_time: 60\nplugin_pipe_size: 102400000\nplugin_buffer_size: 102400\n<\/pre><\/div>\n\n\n\n
After installing and configure with the following nfacctd.conf <\/em>file, you should be able to play with pmacct command to see netflow exports, if you dump results through a pipe, or directly look at the json print export.<\/p>\n\n\n\n
Example of JSON output generated :<\/p>\n\n\n\n
{\n\t\"event_type\": \"purge\",\n\t\"iface_in\": 2,\n\t\"iface_out\": 3,\n\t\"ip_src\": \"207.180.192.205\",\n\t\"ip_dst\": \"192.168.1.101\",\n\t\"country_ip_src\": \"DE\",\n\t\"country_ip_dst\": \"\",\n\t\"tcp_flags\": \"0\",\n\t\"ip_proto\": \"udp\",\n\t\"stamp_inserted\": \"2020-08-20 11:28:00\",\n\t\"stamp_updated\": \"2020-08-20 11:29:01\",\n\t\"packets\": 1,\n\t\"bytes\": 125\n}\n{\n\t\"event_type\": \"purge\",\n\t\"iface_in\": 3,\n\t\"iface_out\": 2,\n\t\"ip_src\": \"192.168.1.101\",\n\t\"ip_dst\": \"5.187.71.162\",\n\t\"country_ip_src\": \"\",\n\t\"country_ip_dst\": \"RU\",\n\t\"tcp_flags\": \"0\",\n\t\"ip_proto\": \"udp\",\n\t\"stamp_inserted\": \"2020-08-20 11:28:00\",\n\t\"stamp_updated\": \"2020-08-20 11:29:01\",\n\t\"packets\": 1,\n\t\"bytes\": 311\n}<\/pre><\/div>\n\n\n\n
Data Export<\/h2>\n\n\n\n
After this you are able to parse data produced by nfacctd collector and push it to a TSDB.<\/p>\n\n\n\n
Python – From JSON to InfluxDB<\/h3>\n\n\n\n
Not sure InfluxDB is the best to doing this, due to the high cardinality on tags. But anyway, for home setup, it’s sufficient.<\/p>\n\n\n\n
I wrote a quick and dirty python script that parse and enhance data (like SNMP ifIndex name provided by nfacctd) and push it to InfluxDB. <\/p>\n\n\n\n
I will probably publish it to github when it will be cleaned up.<\/p>\n\n\n\n
Grafana<\/h3>\n\n\n\n
Here is an example of what you can produce with data inside the TSDB. Top Countries, Top IP talkers… <\/p>\n\n\n\n
\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"
Context Looking on which client in my home network is consuming bandwith and trying to catch \/ debug some attacks and network flows, I played…<\/p>\n
Continue readingExporter les netflows depuis une gateway Unifi<\/span><\/a><\/div>","protected":false},"author":1,"featured_media":2712,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24],"tags":[127,126,125],"class_list":["post-2707","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-reseau","tag-monitoring","tag-netflow","tag-unifi","entry"],"_links":{"self":[{"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/posts\/2707","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/comments?post=2707"}],"version-history":[{"count":4,"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/posts\/2707\/revisions"}],"predecessor-version":[{"id":2714,"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/posts\/2707\/revisions\/2714"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/media\/2712"}],"wp:attachment":[{"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/media?parent=2707"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/categories?post=2707"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beufa.net\/fr\/wp-json\/wp\/v2\/tags?post=2707"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}