Summary: Spectrum “” updated” our DOCSIS cable television modem and it broke all of our IP phones. I found they are rate-limiting incoming port 5060 traffic. Spectrum “” assistance” is reluctant and useless to assist. You may be impacted too. I'' ll reveal you how to check, and how to exploit this vulnerability.

This is a truly long headache of a story, so stick with me.

I am a network engineer with a customer who utilizes IP phones at all of their service places. Last November, almost 4 months earlier, Spectrum came out and changed our old DOCSIS 3.0 cable television modem with a DOCSIS 3.1 modem and router set after we updated the service speed. They set up a Hitron EN2251 cable television modem and Sagemcom RAC2V1S router. Right away later on I began getting problems that phones were not working.

I'' ve separated it down to the cable television modem and/or the service originating from the CMTS/Head Node.

To be technical: Spectrum is rate-limiting all incoming ip4 packages with a source OR location port of 5060, both UDP and TCP. The rate limitation is around 15Kbps and is international to all incoming port-5060 packages transiting the cable television modem, not session or IP-scoped in any method. Outgoing traffic seems untouched. By “” incoming” I suggest from the web to CPE.

I won'' t bore you with the remarkable quantity of effort and time that was taken into separating this issue and fixing, however I wish to make it clear right now that this isn'' t an issue with our firewall program. This isn ' t an issue with the Sagemcom RAC2V1S router either. This is not a SIP-ALG issue.

For those of you who are security mindful and taking note, yes, this is an exploitable vulnerability. Anybody can send out a small quantity of spoofed traffic to any IP behind one of these cable television modems and it will knock out all VOIP services utilizing basic SIP on 5060.

Demonstrating the issue.

Below I run 4 iperf3 tests. I run 2 standard tests coming from port 5061 to reveal what things must look like. I the very same tests however alter the customer source port to 5060. I'' ve supply both the customer and server stdout. The TCP traffic gets restricted down to 14Kbps, and UDP sees 98% package loss. IP addresses have actually been altered for personal privacy.

Test # 1. TCP standard test, traffic untouched.–>> iperf3 -c $IPERF_SERVER -p 5201– cport 5061 -t 10 -b 5M

Client Connecting to host 11.11.11.111, port 5201 [5] regional 222.222.222.222 port 5061 linked to 11.11.11.111 port 5201 [ID] Period Transfer Bitrate Retr Cwnd [5] 0.00-1.00 sec 651 KBytes 5.33 Mbits/sec 0 270 KBytes [5] 1.00-2.00 sec 640 KBytes 5.24 Mbits/sec 0 270 KBytes [5] 2.00-3.00 sec 640 KBytes 5.24 Mbits/sec 0 270 KBytes [5] 3.00-4.00 sec 512 KBytes 4.19 Mbits/sec 0 270 KBytes [5] 4.00-5.00 sec 640 KBytes 5.24 Mbits/sec 0 270 KBytes [5] 5.00-6.00 sec 640 KBytes 5.24 Mbits/sec 0 270 KBytes [5] 6.00-7.00 sec 640 KBytes 5.24 Mbits/sec 0 270 KBytes [5] 7.00-8.00 sec 640 KBytes 5.24 Mbits/sec 0 270 KBytes [5] 8.00-9.00 sec 512 KBytes 4.19 Mbits/sec 0 270 KBytes [5] 9.00-10.00 sec 640 KBytes 5.24 Mbits/sec 0 270 KBytes – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate Retr [5] 0.00-10.00 sec 6.01 MBytes 5.04 Mbits/sec 0 sender [5] 0.00-10.04 sec 6.01 MBytes 5.02 Mbits/sec receiver iperf Done. Server Accepted connection from 222.222.222.222, port 53620 [5] regional 11.11.11.111 port 5201 linked to 222.222.222.222 port 5061 [ID] Period Transfer Bitrate [5] 0.00-1.00 sec 651 KBytes 5.33 Mbits/sec [5] 1.00-2.00 sec 640 KBytes 5.24 Mbits/sec [5] 2.00-3.01 sec 640 KBytes 5.19 Mbits/sec [5] 3.01-4.00 sec 512 KBytes 4.23 Mbits/sec [5] 4.00-5.00 sec 640 KBytes 5.24 Mbits/sec [5] 5.00-6.00 sec 640 KBytes 5.24 Mbits/sec [5] 6.00-7.00 sec 640 KBytes 5.23 Mbits/sec [5] 7.00-8.00 sec 512 KBytes 4.21 Mbits/sec [5] 8.00-9.00 sec 640 KBytes 5.24 Mbits/sec [5] 9.00-10.00 sec 640 KBytes 5.24 Mbits/sec – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate [5] 0.00-10.04 sec 6.01 MBytes 5.02 Mbits/sec receiver

Test # 2. UDP standard test, traffic untouched.–>> iperf3 -c $IPERF_SERVER -p 5201– cport 5061 -t 10 -b 1M -u

Client Connecting to host 11.11.11.111, port 5201 [5] regional 222.222.222.222 port 5061 linked to 11.11.11.111 port 5201 [ID] Period Transfer Bitrate Total Datagrams [5] 0.00-1.00 sec 123 KBytes 1.01 Mbits/sec 87 [5] 1.00-2.00 sec 122 KBytes 996 Kbits/sec 86 [5] 2.00-3.00 sec 122 KBytes 996 Kbits/sec 86 [5] 3.00-4.00 sec 123 KBytes 1.01 Mbits/sec 87 [5] 4.00-5.00 sec 122 KBytes 996 Kbits/sec 86 [5] 5.00-6.00 sec 122 KBytes 996 Kbits/sec 86 [5] 6.00-7.00 sec 123 KBytes 1.01 Mbits/sec 87 [5] 7.00-8.00 sec 122 KBytes 996 Kbits/sec 86 [5] 8.00-9.00 sec 122 KBytes 996 Kbits/sec 86 [5] 9.00-10.00 sec 123 KBytes 1.01 Mbits/sec 87 – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate Jitter Lost/Total Datagrams [5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.000 ms 0/864 (0%) sender [5] 0.00-10.05 sec 1.19 MBytes 996 Kbits/sec 0.138 ms 0/864 (0%) receiver iperf Done. Server Accepted connection from 222.222.222.222, port 53622 [5] regional 11.11.11.111 port 5201 linked to 222.222.222.222 port 5061 [ID] Period Transfer Bitrate Jitter Lost/Total Datagrams [5] 0.00-1.00 sec 117 KBytes 961 Kbits/sec 6603487.927 ms 0/83 (0%) [5] 1.00-2.00 sec 122 KBytes 996 Kbits/sec 25662.928 ms 0/86 (0%) [5] 2.00-3.00 sec 122 KBytes 996 Kbits/sec 100.086 ms 0/86 (0%) [5] 3.00-4.00 sec 123 KBytes 1.01 Mbits/sec 0.650 ms 0/87 (0%) [5] 4.00-5.00 sec 122 KBytes 996 Kbits/sec 0.157 ms 0/86 (0%) [5] 5.00-6.00 sec 122 KBytes 996 Kbits/sec 0.143 ms 0/86 (0%) [5] 6.00-7.00 sec 123 KBytes 1.01 Mbits/sec 0.442 ms 0/87 (0%) [5] 7.00-8.00 sec 122 KBytes 996 Kbits/sec 0.356 ms 0/86 (0%) [5] 8.00-9.00 sec 122 KBytes 996 Kbits/sec 0.218 ms 0/86 (0%) [5] 9.00-10.00 sec 123 KBytes 1.01 Mbits/sec 0.152 ms 0/87 (0%) [5] 10.00-10.05 sec 5.66 KBytes 964 Kbits/sec 0.138 ms 0/4 (0%) – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate Jitter Lost/Total Datagrams [5] 0.00-10.05 sec 1.19 MBytes 996 Kbits/sec 0.138 ms 0/864 (0%) receiver

Test # 3. TCP test, traffic is rate-limited.–>> iperf3 -c $IPERF_SERVER -p 5201– cport 5060 -t 10 -b 5M

Client Connecting to host 11.11.11.111, port 5201 [5] regional 222.222.222.222 port 5060 linked to 11.11.11.111 port 5201 [ID] Period Transfer Bitrate Retr Cwnd [5] 0.00-1.00 sec 76.4 KBytes 625 Kbits/sec 1 18.4 KBytes [5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 0 19.8 KBytes [5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 21.2 KBytes [5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 2 5.66 KBytes [5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 1 5.66 KBytes [5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 1 2.83 KBytes [5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 3 4.24 KBytes [5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 2 5.66 KBytes [5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 4 8.48 KBytes [5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 9.90 KBytes – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate Retr [5] 0.00-10.00 sec 76.4 KBytes 62.6 Kbits/sec 14 sender [5] 0.00-10.04 sec 17.0 KBytes 13.8 Kbits/sec receiver iperf Done. Server Accepted connection from 222.222.222.222, port 53624 [5] regional 11.11.11.111 port 5201 linked to 222.222.222.222 port 5060 [ID] Period Transfer Bitrate [5] 0.00-1.00 sec 4.24 KBytes 34.7 Kbits/sec [5] 1.00-2.00 sec 1.41 KBytes 11.6 Kbits/sec [5] 2.00-3.00 sec 1.41 KBytes 11.6 Kbits/sec [5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec [5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec [5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec [5] 6.00-7.00 sec 4.24 KBytes 34.8 Kbits/sec [5] 7.00-8.00 sec 1.41 KBytes 11.6 Kbits/sec [5] 8.00-9.00 sec 2.83 KBytes 23.2 Kbits/sec [5] 9.00-10.00 sec 1.41 KBytes 11.6 Kbits/sec – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate [5] 0.00-10.04 sec 17.0 KBytes 13.8 Kbits/sec receiver

Test # 4. UDP test, traffic is rate-limited.–>> iperf3 -c $IPERF_SERVER -p 5201– cport 5060 -t 10 -b 1M -u

Client Connecting to host 11.11.11.111, port 5201 [5] regional 222.222.222.222 port 5060 linked to 11.11.11.111 port 5201 [ID] Period Transfer Bitrate Total Datagrams [5] 0.00-1.00 sec 123 KBytes 1.01 Mbits/sec 87 [5] 1.00-2.00 sec 122 KBytes 996 Kbits/sec 86 [5] 2.00-3.00 sec 122 KBytes 996 Kbits/sec 86 [5] 3.00-4.00 sec 123 KBytes 1.01 Mbits/sec 87 [5] 4.00-5.00 sec 122 KBytes 996 Kbits/sec 86 [5] 5.00-6.00 sec 122 KBytes 996 Kbits/sec 86 [5] 6.00-7.00 sec 123 KBytes 1.01 Mbits/sec 87 [5] 7.00-8.00 sec 122 KBytes 996 Kbits/sec 86 [5] 8.00-9.00 sec 122 KBytes 996 Kbits/sec 86 [5] 9.00-10.00 sec 123 KBytes 1.01 Mbits/sec 87 – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate Jitter Lost/Total Datagrams [5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.000 ms 0/864 (0%) sender [5] 0.00-10.05 sec 21.2 KBytes 17.3 Kbits/sec 531773447.595 ms 596/611 (98%) receiver iperf Done. Server Accepted connection from 222.222.222.222, port 53626 [5] regional 11.11.11.111 port 5201 linked to 222.222.222.222 port 5060 [ID] Period Transfer Bitrate Jitter Lost/Total Datagrams [5] 0.00-1.00 sec 4.24 KBytes 34.7 Kbits/sec 1153642567.539 ms 0/3 (0%) [5] 1.00-2.00 sec 1.41 KBytes 11.6 Kbits/sec 1081539952.652 ms 0/1 (0%) [5] 2.00-3.00 sec 2.83 KBytes 23.2 Kbits/sec 950572277.560 ms 47/49 (96%) [5] 3.00-4.00 sec 1.41 KBytes 11.6 Kbits/sec 891161510.925 ms 63/64 (98%) [5] 4.00-5.00 sec 1.41 KBytes 11.6 Kbits/sec 835463917.897 ms 60/61 (98%) [5] 5.00-6.00 sec 2.83 KBytes 23.2 Kbits/sec 734294464.575 ms 126/128 (98%) [5] 6.00-7.00 sec 1.41 KBytes 11.6 Kbits/sec 688401061.323 ms 63/64 (98%) [5] 7.00-8.00 sec 1.41 KBytes 11.6 Kbits/sec 645375997.141 ms 65/66 (98%) [5] 8.00-9.00 sec 2.83 KBytes 23.2 Kbits/sec 567225002.330 ms 121/123 (98%) [5] 9.00-10.00 sec 1.41 KBytes 11.6 Kbits/sec 531773447.595 ms 51/52 (98%) – – – – – – – – – – – – – – – – – – – – – – – – – [ID] Period Transfer Bitrate Jitter Lost/Total Datagrams [5] 0.00-10.05 sec 21.2 KBytes 17.3 Kbits/sec 531773447.595 ms 596/611 (98%) receiver

How can you discover if you are impacted?

It'' s noteworthy that not all Spectrum service appear to be impacted. My client has 2 other places in the exact same city, not even 5 miles away, with Spectrum service, and both of those are untouched by this issue. Those areas have older DOCSIS 3.0 modems (Arris TG862G) on older tradition speed strategies. Bear in mind that we didn'' t have this issue prior to Spectrum came out and changed devices.

Suspected impacted cable television modem designs consist of E31N2V1, E31T2V1, E31U2V1, EN2251, EU2251, es2251, and et2251. These are offered for Spectrum'' s Ultra strategies and anything over 300Mbps.

I'' ve confirmed that a minimum of another Spectrum consumer is impacted, however I put on'' t understand how extensive this is.

To check, you will require to utilize the iperf3 tool to do a rate limitation test.

iperf is offered for Windows, linux, Mac, Android, and more: https://iperf.fr/iperf-download.php

You will require both a customer and server system.

NOTE: If you put on'' t have access to excellent customer system with a public IP address on the web, established your server, leave it up, and send me a PM with your IP address and port. I can run a test versus it and send you the outcomes. Simply utilize some port like 61235 if you are paranoid about security.

The server must live behind the cable television modem being checked. The default port is 5201, however you can utilize any port on the server side as long as'it ' s not 5060. It ' s fine to port-forward the server to a NAT firewall program.

The customer requires to be out on the web someplace and it requires to have a genuine special public IP address. It most likely can'' t lag a NAT firewall software due to the fact that we require to manage the source port it utilizes to send out traffic to the server. Take notice of the customer traffic entering into the server side. If the port gets equated to something besides we define with “”– cport” the test won'' t stand.

The server is truly simple to establish. Simply do “” iperf3 -s” “to begin the server and leave it running. Include “”- p 61235″ to define a various port.

The customer is where the action is. We wish to send out traffic to the server and make certain it'' s got.

Run the following 4 commands on the customer system:

iperf3 -c $IPERF_SERVER -p 5201– cport 5061 -t 10 -b 5M

iperf3 -c $IPERF_SERVER -p 5201– cport 5061 -t 10 -b 1M -u

iperf3 -c $IPERF_SERVER -p 5201– cport 5060 -t 10 -b 5M

iperf3 -c $IPERF_SERVER -p 5201– cport 5060 -t 10 -b 1M -u

– c is for the customer IP. change the $IPERF_SERVER with your server public IP. -p is the server port and ought to match the server, the default is 5201. -t is length of test, 10 seconds. -b is bandwidth, restricted to 5Mbps for TCP and 1Mbps for UDP. -u is a UDP test, rather than the default TCP.

— cport is the customer traffic source port, and this is where the magic occurs. I'' m utilizing port 5061 as a standard measurement port, which need to be untouched by any rate limitation, however you might utilize anything aside from 5060.

It'' s typical to see some little<( <5% )package loss on the UDP tests. Wear'' t concern if you can'' t get 5Mbps on the TCP test. Simply take note the distinction in between utilizing port source port 5060 and anything else.

If Spectrum is rate-liming your traffic, you will see a significant distinction in the outcomes. You may see 100Mbps on the port 5061 test and after that less than 20Kbps on the 5060 test. On UDP you would see almost 0% package loss on the UDP standard test and>> 80% loss on the 5060 test.

Q: If this issue was extensive, other individuals would have discovered?

This is the huge concern I have today. Why are we are impacted, and who is else out there impacted? You would believe that individuals would see if all of their SIP phones quit working, however it ends up the rate limitation is simply high enough to let a couple of phones through without problem. It'' s possible this issue is restricted to specific accounts, or possibly it'' s local, the head node/CMTS, or possibly other clients put on'' t have adequate phones to see.

'I ' ve discovered another client who can recreate the issue, so I understand it'' s not simply us.

My screening reveals I can get up to 7 of our Yealink phones signed up with the SIP server, as long as I stagger their preliminary connections. With less than 4 phones I can'' t set off the problem at all since there'isn ' t adequate SIP traffic. Anything past 10 phones triggers all of them to continuously lose their registration. The more phones, the more SIP traffic, and the even worse the issue gets.

Most clients most likely wear'' t have as numerous phones as we do, and this issue just appears to be impacting the more recent cable television modems and higher-tier service, and not all VOIP service providers utilize ports 5060 for their signaling traffic. Yes, It'' s possible this is a nationwide problem and no one has actually seen or been able to figure out what'' s going on here.

Q: So why would Spectrum be doing this? What'' s their intention?

I believe the response may be right here:

DDoS Attacks: VoIP Service Providers Under Pressure

Phone calls interfered with by continuous DDoS cyber attack on VOIP.ms

I believe this may be some sort of moron'' s Denial of Service policy failed.

Spectrum has an item requirements sheet here that mentiones “” Security • DOS (rejection of service) attack security””.

Back in late September of 2021, practically 30 days prior to this issue began, a variety of VOIP server/carriers were struck with big DDoS attacks. My customer'' s phones were impacted by this attack too, and we saw, however it just lasted a number of days and after that the attack was reduced.

It'' s possible Spectrum was attempting to reduce or avoid reflection attacks versus their consumers, or possibly they are being anti-competitive and attempting to require consumers into utilizing their own VOIP services Who understands and I wear'' t care.

It ' s notable that the modem likewise limits the quantity of ICMP traffic it produces (non transit) so greatly that 2 MTR sessions will trigger it to begin dropping packages. If they are dumb sufficient to do that, then I can see them fucking with other types of traffic.

All other traffic appears to be untouched, as far as I understand, however I wouldn'' t be stunned to discover something else is restricted. I did check a number of ports typical to reflection attacks such as 53 and 123 however they showed up unfavorable.

Testing approaches and other info.

This isn'' t an issue with any IP allotment, though I didn ' t test ipv6. We get a/ 29 from Spectrum, however if you plug straight into the cable television modem you can get a public-unique IP address from an entirely various subnet through DHCP, however the issue continues. Altering your CPE MAC address triggers a brand-new IP address to be assigned, so it'' s simple to check various addresses. This likewise makes it clear the issue isn'' t the Sagemcom RAC2V1S router that Spectrum mandates we utilize for the IP allowance.

I'' m relatively particular this isn'' t a SIP-ALG service in the cable television modem, however that'' s possible. The material of the packages doesn'' t matter, and I can'' t discover any proof that SIP traffic is really being changed in any method, even after attempting. Both MonsterVOIP and RingLOGIX have SIP-ALG test tools and those pass since they wear'' t send out enough traffic to activate the rate limitation.

We'' ve removed all other possibilities at this moment. We evaluated 4 various firewall softwares and linux boxes behind the modem. The truth that we have other Spectrum areas in the exact same city to check from, simply miles away, suggests we eliminated a 3rd celebration transit company too. There'' s actually absolutely nothing left however Spectrum to blame here.

What about Intel Puma chipsets?

While investigating this issue I discovered everything about the concerns with Intel Puma chipsets in DOCSIS cable television modems. I truly wear'' t understand if this is the source of issue or if this is some sort of policy administratively enforced.

Apparently there are just 2 DOCSIS 3.1 chipsets presently on the marketplace, the Intel Puma 7 (Intel FHCE2712M) and the Broadcom BCM3390 .

The older Intel Puma 6 chips are very popular for being dreadful . There are many short articles recording all of the modems they remain in, and which to prevent. There'' s been class action suits. To state they are bad is an understatement . Obviously the more recent Puma 7 chips still have latency issues.

We'' ve had a Hitron EN2251 and a Sercomm ES2251 set up and both of those modems absolutely have an Intel Puma 7 chipset. We just recently got a Technicolor ET2251 set up, and that'' s expected to possibly have a Broadcom chip. The port 5060 restricting continues.

There are some reports that the Technicolor and Ubee versions of these modems might have the Broadcom chip, however other reports state the more recent systems after 2018 have Intel Puma chips too, and I simply wear'' t understand what the reality is. This customer is far away so I can'' t simply take a screwdriver and break the case to discover out.

Note that my customer has an organization account and Spectrum will never let us utilize our own cable television modem. They mandate that they provide the modem , and since we have fixed IPs, they provide us that dumb Sagemcom router too. I'' ve made efforts to obtain our own provided modem however no one at Spectrum will enable it. Both Spectrum'' s dispatch techs and assistance representatives state that you can'' t demand particular hardware when asking for a modem swap which you get whatever the storage facility sends out and you'' ll like it.

What to do?

There is definitely no validation for Spectrum to be fucking with our SIP traffic like this, or any other traffic.

To work around this problem I merely routed the SIP traffic out over a VPN tunnel to among our other neighboring places, which likewise has Spectrum service, which makes the issue disappear. In the long term I put on'' t desire to do foolish workarounds like this.

If our VOIP service provider supported service utilizing a port besides 5060 we might alter the phones to utilize that, however they put on'' t. We prepare to ditch our present service provider in the next year anyhow, so that'' ll most likely look after the issue too.

Beyond the above, we currently have some attorney letters heading out to the FCC and state federal government. If I can'' t get anybody at Spectrum with 2 brain cells to rub together here quickly, we will sue in little claims court, which is something I'' ve done a number of times previously, and it'' s really efficient. When the business workplace legal representatives get included and they need to send out a worker to court, shit gets repaired genuine quickly.

But I'' m certainly open up to recommendations.

Oh yea, practically forgot, click on this link for a great time.

sent by / u/Cheeseblock27494356 [link] [remarks] .

Read more: reddit.com

Leave a comment

Your email address will not be published.