The router is a second-hand product that I purchased from XianYu (Chinese clone of eBay) on April 18, 2023. The specifications are roughly as follow:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
Product code RB4011iGS+RM
Architecture ARM 32bit
CPU AL21400
CPU core count 4
CPU nominal frequency auto (533 - 1900) MHz
Switch chip model RTL8367SB
Dimensions 228 x 120 x 30 mm
RouterOS license 5
Operating System RouterOS (v7 only)
Size of RAM 1 GB
Storage size 512 MB
Storage type NAND
MTBF Approximately 200'000 hours at 25C
Tested ambient temperature -40°C to 70°C
IPsec hardware acceleration Yes
Suggested price $219.00
|
The reason for buying such a router is simple:
I’ve been using virtualized Linux routers before, which have many advantages like outstanding performance and flexible.
However, maintaining the hypervisor is very difficult, any hardware adjustments can cause the entire network to go down, which can annoy the roommates if it happens frequently.
If I want to use Proxmox’s HA for hot migration and high availability, I’d need to set up a Ceph storage service and two new EPYC servers,
which would be too expensive in terms of hardware and running costs, in that case it’s better to get a separate 1U Celeron server as a router.
Hardware requirements
I myself haven’t paid much attention to consumer-grade wireless routers for a while, so I won’t make any horizontal comparisons for now.
As for my own needs, besides the standard RJ45 GbE interface, I also need at least one SFP+ cage for future-proofing.
ISPs in Shanghai have basically rolled out 1000Mbps broadband, and considering the low update frequency of routers, it shouldn’t be too extravagant to buy a router that supports 10Gbps interfaces in preparation for future gigabit or higher broadband.
This device supports 10 Gigabit Ethernet ports and one SFP+ cage, which meets my needs perfectly. It is also equipped with a sufficiently powerful 1.9GHz ARM processor that can handle my requirement of running 500Mbps on WireGuard.
But it should be noted that the 10 Ethernet interfaces of this router are not all directly connected to the switch chip/CPU. Instead, they are divided into two groups, each connected to a 2.5G switch chip and then connected to the CPU. You can refer to this diagram:
In other words, when connecting devices, it is necessary to avoid accessing large flow across switches as much as possible, otherwise it may hit the 2.5Gbps communication limit of the switch chip. However, this issue can be ignored for me. My anticipated usage scenario is to connect the ISP cable to ether1 and the ZTE switch to SFP+. The local network traffic will not pass through the router, so it will not hit this limit at all.
Software Configuration
Here things here are starting to get messy, because my needs are really messy.
- BGP support is required: I have my own Autonomous System Number (ASN) and a publicly reachable IPv6 prefix. I need to establish BGP sessions and advertise them in Hong Kong and to upstream providers. Each router in different regions will use private ASN in a large internal network for routing and BGP will be used as the routing protocol.
- WireGuard support is required: My internal network primarily uses WireGuard for communication. It would be inconvenient if this device does not support WireGuard (IPsec and OpenVPN are not as user-friendly as WireGuard, and I believe most people would agree on this).
- Performance should be sufficient to accept a full BGP routing table from external sources for proxy routing: This is well understood, older devices may not be able to handle this.
- Basic features such as VLAN, VRF, and policy-based routing should also be supported.
Fortunately, RouterOS v7 can basically meet all the above requirements.
Upstream Network Configuration
Let’s create a PPPoE client for internet first:
1
2
3
4
5
6
7
|
[admin@MikroTik] > /interface/pppoe-client/print detail
Flags: X - disabled, I - invalid; R - running
0 R name="pppoe-chinanet" max-mtu=auto max-mru=auto mrru=disabled
interface=ether1 user="redacted" password="redacted" profile=default
keepalive-timeout=60 service-name="" ac-name="" add-default-route=yes
default-route-distance=100 dial-on-demand=no use-peer-dns=no
allow=pap,chap,mschap1,mschap2
|
Then create one IPv6 DHCPv6 client to obtain IPv6 prefixes from ChinaNet:
1
2
3
4
5
6
7
8
9
|
[admin@MikroTik] > /ipv6/dhcp-client/print detail
Flags: D - dynamic; X - disabled, I - invalid
0 ;;; ChinaNet DHCPv6 PD
interface=pppoe-chinanet status=bound duid="redacted"
dhcp-server-v6=fe80::ce1a:faff:feeb:d1c0 request=prefix
add-default-route=no use-peer-dns=no dhcp-options=""
pool-name="chinanet-ipv6-pd" pool-prefix-length=56 prefix-hint=::/0
script=redacted
dhcp-options="" prefix=240e:redacted::/56, 32m8s
|
Thus the basic internet access is almost okay.
WireGuard Configuration
The configuration style of WireGuard in RouterOS is different from other systems, they separates the configuration of interfaces and peers.
When adding peers, you need to explicitly specify an interface.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[admin@MikroTik] /interface/wireguard> print detail where name=sddn-unicom-cz
Flags: X - disabled; R - running
2 R name="sddn-unicom-cz" mtu=1400 listen-port=redacted
private-key="redacted"
public-key="redacted"
[admin@MikroTik] /interface/wireguard> peers print detail where interface=sddn-unicom-cz
Flags: X - disabled
0 interface=sddn-unicom-cz
public-key="redacted"
endpoint-address=redacted endpoint-port=redacted
current-endpoint-address=redacted current-endpoint-port=redacted
allowed-address=0.0.0.0/0,::/0 persistent-keepalive=25s rx=830.5MiB
tx=507.9MiB last-handshake=1m5s
|
Since the interface configuration is complete,
we need to add v4 and v6 addresses to our interface configurations on /ip/address and /ipv6/address respectively.
Please note that RouterOS is not very friendly when establishing BGP sessions using IPv4 addresses with /32 prefix.
It’s recommended to use /31 or larger interface addresses for further BGP configurations.
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[admin@MikroTik] /routing/bgp> /ip/address/print where interface=sddn-unicom-cz
Columns: ADDRESS, NETWORK, INTERFACE
# ADDRESS NETWORK INTERFACE
;;; IPv4 Address for BGP Session with Unicom-Changzhou
2 28.0.15.9/30 28.0.15.8 sddn-unicom-cz
[admin@MikroTik] /routing/bgp> /ipv6/address/print where interface=sddn-unicom-cz
Flags: D - DYNAMIC; G, L - LINK-LOCAL
Columns: ADDRESS, INTERFACE, ADVERTISE
# ADDRESS INTERFACE ADVERTISE
;;; IPv6 Address for BGP Session with Unicom-Changzhou
1 G 2001:db8:reda:cted::/126 sddn-unicom-cz no
2 DL fe80::6b4e:4fe3:b34:cf1c/64 sddn-unicom-cz no
|
BGP session for intranet
Just like using BIRD, it is best for us to prepare a template first.
1
2
3
4
5
|
[admin@MikroTik] /routing/bgp> template/print
Flags: * - default; X - disabled, I - inactive
0 * name="default" routing-table=main router-id=28.1.1.1 as=4233331011
multihop=no address-families=ip,ipv6
output.redistribute=connected,static
|
Then create BGP configurations for v4 and v6 using these two templates.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
[admin@MikroTik] /routing/bgp> connection/print where name="bgp-sddn-unicom-cz-v4" or name="bgp-sddn-unicom-cz-v6"
Flags: D - dynamic, X - disabled, I - inactive
0 name="bgp-sddn-unicom-cz-v6"
remote.address=2001:db8:redacted::a .as=4233330010
local.address=2001:db8:redacted::9 .role=ebgp
routing-table=main router-id=28.1.1.1 templates=default as=4233331011
nexthop-choice=default multihop=no address-families=ipv6
output.redistribute=connected,static,dhcp .filter-chain=sddn-export
.no-early-cut=no .keep-sent-attributes=no
input.filter=sddn-cn-import
1 name="bgp-sddn-unicom-cz-v4"
remote.address=28.0.15.10 .as=4233330010
local.address=28.0.15.9 .role=ebgp
routing-table=main router-id=28.1.1.1 templates=default as=4233331011
multihop=no address-families=ip
output.redistribute=connected,static .filter-chain=sddn-export
input.filter=sddn-cn-import
|
Of course we may also need to add some filters to make some adjustments to the routing priority as needed.
1
2
3
4
5
6
7
|
[admin@MikroTik] /routing/bgp> ../filter/rule/print where chain=sddn-cn-import
Flags: X - disabled, I - inactive
0 chain=sddn-cn-import
rule="if (afi ipv4) {\r\n set pref-src 28.0.99.11;\r\n}\r\n\r\nif (afi
ipv6) {\r\n set pref-src 2001:db8:redacted::28.0.99.11;\r\n}\r\n\r\nif
(bgp-large-communities includes 142641:100:2) {\r\n set distance
+5;\r\n} else {\r\n set distance -5;\r\n}\r\n\r\naccept"
|
In addition to the WireGuard + BGP connection to one of my servers mentioned above, I have also configured a connection to Hong Kong and Japan. I will not go into further details here.
Bring BGP full-table back to home
The easiest way to obtain a full table solution currently is still through Vultr’s cheapest VPS. There are many articles available online for reference, and those who need it will naturally proceed with the operation. However, my implementation strategy on RouterOS is rather mysterious: I use BGP to accept non-mainland IPv4 routes and mainland IPv6 routes.
Regarding the IPv4 part, I plan to use the gateway obtained through PPPoE to access the Internet directly. I want to pull non-mainland routing prefixes into the main table and set them up as an ipip tunnel interface within the internal network by using a filter rule. Since the gateway (peer) is dynamic after the PPPoE connection is established, I prefer to avoid using this parameter as much as possible.
1
2
3
4
5
|
[admin@MikroTik] /routing/bgp> ../filter/rule/print where chain=bgp-speaker-proxy-v4
Flags: X - disabled, I - inactive
3 chain=bgp-speaker-proxy-v4
rule="if (dst-len > 0) {\n set gw 28.0.15.14;\n set gw-interface ipip-
internet-gateway;\n accept;\n} else {\n reject;\n}"
|
For IPv6, the router in my intranet has announced a default route, which is dynamic. Fortunately, the link-local addressing mechanism of IPv6 allows me to use a stable fe80::xxxx%pppoe-chinanet gateway as an exit, which is very fixed. That’s why I used the opposite mode compared to v4:
1
2
3
4
5
6
|
[admin@MikroTik] /routing/bgp> ../filter/rule/print where chain=bgp-speaker-direct-v6
Flags: X - disabled, I - inactive
4 chain=bgp-speaker-direct-v6
rule="if (dst-len > 0) {\r\n set gw fe80::redacted;\r\n set gw-interface
pppoe-chinanet;\r\n set distance -5;\r\n accept;\r\n} else {\r\n reject;
\r\n}"
|
Let’s test the routing effect of the router using 1.1.1.1, 223.5.5.5, 240e:96c:6100:b15:3e::, and 2001:470:0:1f9::2 separately:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
[admin@MikroTik] /routing/bgp> /ip/route/print where 1.1.1.1 in dst-address and
routing-table=main
Flags: D - DYNAMIC; A - ACTIVE; b, v, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAv 0.0.0.0/0 pppoe-chinanet 100
DAb 1.1.1.0/24 28.0.15.14%ipip-internet-proxy 20
[admin@MikroTik] /routing/bgp> /ip/route/print where 223.5.5.5 in dst-address and routing-table=main
Flags: D - DYNAMIC; A - ACTIVE; v, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAv 0.0.0.0/0 pppoe-chinanet 100
[admin@MikroTik] /routing/bgp> /ipv6/route/print where 240e:96c:6100:b15:3e:: in dst-address and routing-table=main
Flags: D - DYNAMIC; A - ACTIVE; b, v, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAb ::/0 2001:db8:redacted:cc23::6 20
D b ::/0 2001:db8:redacted:cc23::a 25
D v ::/0 pppoe-chinanet 100
DAb 240e::/20 fe80::ce1a:faff:feeb:d1c0%pppoe-chinanet 15
DAb 240e:96c::/30 fe80::ce1a:faff:feeb:d1c0%pppoe-chinanet 15
DAb 240e:96c::/32 fe80::ce1a:faff:feeb:d1c0%pppoe-chinanet 15
[admin@MikroTik] /routing/bgp> /ipv6/route/print where 2001:470:0:1f9::2 in dst-address and routing-table=main
Flags: D - DYNAMIC; A - ACTIVE; b, v, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAb ::/0 2001:db8:redacted:cc23::6 20
D b ::/0 2001:db8:redacted:cc23::a 25
D v ::/0 pppoe-chinanet 100
|
It can be seen that basically, the expected correct routing is met.
IPv6 Prefix Address Translation
At this moment, my router has both an IPv6 prefix from China Telecom and an IPv6 prefix that I obtained from my own LIR.
However, because the LAN side uses SLAAC to distribute its own prefix, even though I have performed traffic engineering using the complete table data, the source address still seems off:
the Telecom interface receives a packet with an address that is completely different from the one it distributed.
Interestingly, Telecom does not filter this type of traffic but correctly forwards the packet using its own network to the final destination.
When the target server sends a response, it “conforms to expectations” by using the source address from the previous packet as the destination address.
Since this address is advertised by my ASN in Hong Kong, it will be routed to routers in Hong Kong and then transmitted through my intranet to my home, resulting in poor network performance as the network packets travel globally.
The solution to this problem is quite simple, we just need to add a little magic to the IPv6 firewall and perform NAT for the traffic passing through the China Telecom gateway.
1
2
3
4
5
6
7
8
9
|
[admin@MikroTik] /routing/bgp> /ipv6/firewall/nat/print
Flags: X - disabled, I - invalid; D - dynamic
0 chain=dstnat action=netmap to-address=2001:redacted::/56
dst-address=240e:redacted::/56 in-interface=pppoe-chinanet log=no
log-prefix=""
1 chain=srcnat action=netmap to-address=240e:redacted::/56
src-address=2001:redacted::/56 out-interface=pppoe-chinanet log=no
log-prefix=""
|
Of course, because these rules are written with the prefix of the router’s telecommunication, we need to make some adjustments to the previous DHCP client. We can use a script to dynamically update these two firewall rules.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
:log info ("dhcpv6 client script started");
:local ChinaNetPrefix [/ipv6/pool/get [find name="chinanet-ipv6-pd"] prefix];
:log info "prefix assigned by ChinaNet is: $ChinaNetPrefix";
:local ChinaNetPPPLinkLocalAddress [/ipv6/dhcp-client/get [find pool-name="chinanet-ipv6-pd"] dhcp-server-v6];
:log info "link local address ChinaNet is: $ChinaNetPPPLinkLocalAddress";
/routing/filter/rule/set rule="if (dst-len > 0) {set gw $ChinaNetPPPLinkLocalAddress;set gw-interface pppoe-chinanet;set distance -5;accept;} else {reject;}" [find chain="bgp-speaker-direct-v6"];
:log info "filter update okay";
:local ChinaNetNewPrefix [:pick "$ChinaNetPrefix" 0 ([:len "$ChinaNetPrefix"] - 3)]
/ipv6/firewall/nat/set dst-address="$ChinaNetNewPrefix/56" [find chain=dstnat action=netmap];
/ipv6/firewall/nat/set to-address="$ChinaNetNewPrefix/56" [find chain=srcnat action=netmap];
:log info "netmap update okay";
|
Multiple VLANs and Acceleration of Game/Steam downloads
The solution for these issues is relatively consistent, which is to establish multiple different routing tables, set up respective routes in the tables, and associate the configuration of each VLAN interface with these tables using /routing/rule.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
[admin@MikroTik] /routing/bgp> /routing/rule/print
Flags: X - disabled, I - inactive
0 src-address=10.0.0.0/24 interface=wg-as4134 action=lookup
table=egress-pppoe
1 interface=vlan-gaming-hk action=lookup table=gaming-hk
2 interface=vlan-gaming-jp action=lookup table=gaming-jp
[admin@MikroTik] /routing/bgp> /ip/firewall/mangle/print where chain=prerouting
Flags: X - disabled, I - invalid; D - dynamic
0 D ;;; special dummy rule to show fasttrack counters
chain=prerouting action=passthrough
5 ;;; allow Hong Kong gaming VLAN accessing other intranet
chain=prerouting action=mark-routing new-routing-mark=main
passthrough=no src-address=28.1.4.0/24 dst-address=28.0.0.0/8
in-interface=vlan-gaming-hk log=no log-prefix=""
6 ;;; allow Japan gaming VLAN accessing other intranet
chain=prerouting action=mark-routing new-routing-mark=main
passthrough=no src-address=28.1.5.0/24 dst-address=28.0.0.0/8
in-interface=vlan-gaming-jp log=no log-prefix=""
7 ;;; HKRPG overseas UDP
chain=prerouting action=mark-routing new-routing-mark=gaming-jp
passthrough=no protocol=udp dst-port=23301 log=no log-prefix=""
|
Some Other Issues Occurred
The implementation of ipip/gre tunnel in RouterOS refers to Cisco’s keep alive implementation. When used with Linux devices, it is unable to determine if the other party is online. You can check the specific reason here: Replying to GRE keep alive packets on Linux
The above is about the situation of using RB4011 in my own network environment. In actual use, there are hardly any problems except for occasional strange issues with RouterOS. Finally, here is a screenshot of WebFig:
