Determine regulating mechanism for Network Bandwidth and Congestion

[Pedro-on-a-bike]

^{This initial comment is collaborative and open to modification by all.}

Task Summary

There is currently no defined mechanism to regulate bandwidth and therefore alleviate congestion on the network. There is also no mechanism to define and discriminate traffic. The network is currently unrestricted and will assume full bandwidth for all users until saturated.

Moving forward, say if we are going to adopt a model where we suggest a donation to help support the network, are we then going to have a blanket QoS? Or traffic shape all connections to 15Mbps? What happens if one node requires more bandwidth than another? How do we prioritize traffic? In addition, how do we disincentivize users from saturating bandwidth and just leaving YouTube on all day at 4K. Do we then switch to PPPoE and define a tiered model? Or do we use a pay-per-use model to achieve the same behaviour?

🎟️ Re-ticketed from: #
📅 Due date: N/A
🎯 Success criteria:

[MAIN TEXT HERE]

To Do

• Determine the long term plan as to how we will regulate bandwidth on the network. Will it be a self-regulating pay-per-use model, will it be a subscription model for tiered service through PPPoE, a donation model and use QoS to treat all traffic the same?
• If we use a pay-per-use model, fees collected can still be redirected back to organization to help pay for overhead and infrastructure. A pay-per-use model would be similar to cell phone plans with the desired effect on end user behaviour so they are price conscious of their usage. This could also allow the network to easily grow as it will be self regulating by only requiring each individual to monitor and adjust their behaviour. A primary con is that pay-per-use is generally frowned upon given their abuse in cell phone plans, however fees on this network can be potentially set at fraction of the price to make it affordable.
• If we use a subscription model for tiered service via PPPoE, we essentially are becoming an WISP. The big issue with this is that it now centralizes all access to the org and increases administration and governance for anyone who wants to join the network. It does however allow for discrimination if one node requires more bandwidth vs another. This can also potentially limit organic growth of the mesh as each new node will need to request access from the central org, as opposed to a pay-per-use model would only need to request access from their neighbor.
• If we are looking at a donation model and use QoS or blanket traffic shaping for all users, this will be very simple to implement, however it will become an issue if you have say have a community node that doesn't have direct access to a gateway and only has access through another node. If that other node is already limited to a 15Mbps connection, then that secondary node will also be capped at 15Mbps to service all their clients. This model will also not address the issue if different users have different bandwidth needs.

[darkdrgn2k]

I think it may be to early to select a traffic control schema because we have no traffic and therefore don't really know what we need to do.

However I'm listing several traffic shaping options and as i understand them pros/cons. Not a definite list and some information may be inaccurate. Please add/correct as needed!

In a large network EACH DEVICE must follow the same traffic control standard for it to be effective. This is a large challenge with traffic shaping is the mesh topology. This is also the same reason QoS over the internet will not work. However it can still be used by sections of the mesh to deal with transit over their little part.

Traffic Shaping Options - Overview

UNMS

UNMS allows you to control the flow of traffic on Ubiquiti devices using their proprietary interface. Manipulates devices to facilitate the flow.

PRO

• Seems to be easy to setup
• Click and point setup

Con

• Works ONLY on ubiquiti hardware
• Not sure how it handles non-ubiq foreign networks
• Not sure of additional CPU cost for enabling it on resource strapped equipment.
• Only way to expose UI for such management. Local UI on device does not support this.

PPPoE

Used by many isps in Ontario. Creates a tunnel over LAYER 2 back to ISP controlled server. Traffic shaping added there

Pro

• Control of the connection from Peer to Edge
• Built into all routers

Con

• ❌ Requires LAYER 2 network (Point to Point over ETHERNET)
• traffic control happens on both ends only, not in "the middle". Used more to set overall speeds.
• May affect hardware acceleration of routers as it encapsulates like L2TP

TC

TC is Linuxes traffic control/QoS interface (like iptables is for firewalls). It support many different mechanisms for different applications.

Pro

• Exists on probably all Linux distros
• very flexible in what it can do
• Modules available to do things "better"

Con

• Possibly could add CPU cost on resource strapped equipment.
• Needs access to linux shell
• Can be very complicated
• not all modules may be avialble everywhere

---

TC Modules

Class Based Queuing - CBQ

Hierarchical Token Buckets - HTB

https://www.linuxjournal.com/article/7562

HTB is meant as a more understandable, intuitive and faster replacement for the CBQ qdisc in Linux. (lower CPU usage & better help). Both CBQ and HTB help you to control the use of the outbound bandwidth on a given link. Both allow you to use one physical link to simulate several slower links and to send different kinds of traffic on different simulated links. In both cases, you have to specify how to divide the physical link into simulated links and how to decide which simulated link to use for a given packet to be sent.

CoDel

CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and RTT”, and simple AQM algorithm.

Common Applications Kept Enhanced - CAKE

Cake is the rollup of 3 years of deployment experience of the htb + fq_codel based sqm-scripts SQM for aqm/fq/qos inbound and outbound bufferbloat management.

https://www.bufferbloat.net/projects/codel/wiki/Cake/

[darkdrgn2k]

I think before most of these questions can be answered we need to figure out how to actually onboard a person. What is required, what kind of experience do they even get being alone on the network.

I think initially a blanket suggested donation is the easiest way to get things started. We can update the ToS to include some verage about maintaining by use of traffic control or something.

Worst case scenario if we need to move to a more specific method we can at WORST grandfather the early adopters before we piviot to a different model.

[Pedro-on-a-bike]

To add to this, based on other discussions we have had and testing, the simplest solution right now to avoid congestion and make sure bandwidth is available to all members is to add a traffic shaping policy on all CPEs to limit the TX and RX rates to 25Mbps for individual users and adjust as needed and limit the total number of users that have access to a specific Access Point. At this current time, we are using Ubiquiti equipment for all access points and the CPEs require credentials to establish a connection over AirMax and this can be managed through the MAC Address allowed list as no (LAP-Lite) Access Point should have more than 20 CPEs assigned to it. Since both the network administrations and the end user will need remote access to the CPE, it is probably easiest at this time just to have a terms of use and "honor" system for the time being that all CPEs on Point to Multi Point links must have some sort of traffic shaping policy until we establish a long term plan for our "terms of use" policy.