In the past year I have come at heads with Virgin Media regarding their fair use policy, which is basically “use all you want at peak times unless you start to slow down other people’s connections, in which case we give you a warning”. Apparently I was downloading 60GB non-stop every hour (let’s ignore the fact that isn’t possible on my connection speed for a moment). After frustration that there is no decent persistent bandwidth logging on the router (it gives you the amount you’ve uploaded and downloaded since the last boot) I wrote a python script that polls the state if the router every minute to check these figures, calculate a delta for how much was transferred in the last minute and store that to a database. Now this script works (I’ve been using it since July) and I am now releasing it for others to use/improve.
Firstly, anyone who looks at the structure of the script will notice that it was built to function, not to be beautiful. This tool was built so that I could get a log as soon as possible, so I took some very unusual turns in it’s design. You will see that I am storing a string for all the data that I was collecting from the web server in the database every minute. This is because at the original creation of the script I was unsure whether the router counted in a decimal (i.e. 1000K in 1M) or binary (i.e. 1024K in 1M) fashion. Turns out it’s using the latter so I could trim that out. This data is also stored because at some point I intended to add a check function that would make sure all the deltas are correct, but that will be added later.
Writing this script has allowed me to find one bug in the routers firmware. At seemingly random points (but in my own experience this occurs after an automatic reboot), the router works as intended but something occurs to cause it to return 4GB for one or both of the two figures. As there is no remote reboot feature on this router, I have to manually reset it, though I may make something that allow me to trigger a reboot on detection of this condition. If I do that, I will document it here.
Originally this script sat in my ~/bandwidth directory on my server, logged to a file called bandwidth.txt and was called once a minute, with a support script being called at midnight that read the entire file, summing up every hour/day, and emailing the summed results for the past 24 hours and 1 day. As you can imagine over time this got slower and slower. It got to a point at the end where it would take almost 2 minutes to generate the report, at which point something had to be done. Which is where the MySQL conversion comes in.
The database was added as an easier store (its easier to request the last line from the database than read in the entire file and discard old results) as well as add more flexible reporting to the script. This conversion involved adding just the previously stored file data, with optimisation coming later. I have been able to create in addition a page that shows the live reports as a webpage as a result of this page. I haven’t added this to the source yet but I will tweak it to make the code look better before adding. You can see my version running on http://home.simoncook.org/bandwidth.
My current Internet usage statistics
This is where the project is. It is functional, but not elegant, and only works with one router, though I feel that it should be easily expanded as the web page scraping is done in one function, so anyone wanting to create a modified version for their router only needs to create an output-compatible version of this function and they’re good to go. I will modify this script over time both in terms of tidying it up and adding new features that I feel are of use.
The URL for the source for this project is at https://github.com/simonpcook/bandwidth-VMNG480 and if you feel you can make use/make changes for it, feel free to fork it to meet your own needs. I will flesh out some documentation soon (and the required schema will be added to the repo). I hope you find this of use.
Disclaimer: Using this script counts as the single user who can log in. Attempting to login to the router from another computer will give an error message, the current workaround is to administer the router from the computer running the script (or some equivalent such as using a proxy so the router sees the correct source IP) or to disable the script and wait for the login timeout when you want to administer the router.