A flood with no attacker

In May 2003, the University of Wisconsin-Madison started taking hundreds of megabits per second of inbound traffic aimed at one public time server. Nobody was attacking them.

“the root cause is actually a serious flaw in the design of hundreds of thousands of one vendor's low-cost Internet products”

Tuesday normal, Wednesday chaos

Campus inbound traffic peaked around 40,000 packets per second. At 8AM on May 14 it spiked, and by 9:40AM it was breaking measurement gear and older campus routers.

One weird tell

The packets were 76-byte UDP queries to port 123. All of them, from thousands of different addresses, carried the same source port: 23457.

“(Note that the number 23457 seems hand-picked, as the number subsequent to 23456.)”

Block one port, keep serving

Because every flood packet shared that source port, staff could drop just UDP from port 23457 upstream at WiscNet's border, while real NTP clients kept getting answers.

What SNTP is for

SNTP is a stripped-down NTP: send one small UDP packet, read the transmit timestamp in the reply, set your clock. It exists so a cheap device can know the time without running full NTP.

A month later, worse

By early June the blocked traffic was still climbing, holding above 250,000 packets per second and 150 megabits per second. The dips in the graph were maintenance, not relief.

The packets were legitimate

Letting a trickle through revealed well-formed SNTP version 1 queries. Nothing malformed, nothing malicious. The only insane part was the rate: about one query per second, per device.

“One query per second is ridiculous, and is far from best practice for NTP client behavior.”

Real hosts, not spoofs

The source addresses resolved to DNS names and matched the interfaces they arrived on. These were genuine machines around the world, not forged headers from one attacker.

Asking other campuses

None of the sources were local, so Plonka emailed abuse teams at two universities with packet logs and asked what was bound to port 23457. Both came back with the same answer: a Netgear router.

Found with the strings command

He downloaded Netgear firmware and ran strings on it. Out came port 23457, and out came 128.105.39.11, the university's own time server, baked into the binary.

“$ strings MR814_4_11.bin |grep 23457”

The flaw in three lines

The client hard-coded one server IP, hard-coded one source port, and polled every second until it got a reply. A device that never got an answer would hammer forever.

Netgear support ignored it

The June 16 report went nowhere. Plonka called headquarters, left voicemails for executives, and guessed their email addresses. Support's canned reply arrived 23 days later.

“Your issue may have already been resolved. Please reply to this email if you still require assistance”

A review team, not a lawsuit

Once a director finally called back, the two sides built a fifteen-person review team: Netgear staff, university staff, and outside experts from the registries, measurement research, and NTP itself.

Counting the swarm

More than 500,000 unique Netgear addresses hit the server in a single day, an undercount thanks to NAT and dropped broadband links. Netgear reported 707,147 affected units built.

The worst case, in numbers

700,000 devices at one query per second is 700,000 packets per second, or 426 megabits per second of pure time requests pointed at one machine.

Rules for a polite client

The team wrote out what SNTP clients should do: poll between 64 and 1024 seconds, resolve names by DNS before each query, back off on failure, and never speed up when a server stops answering.

“MUST NOT use a shorter poll interval upon failure to receive a response”

The fix shipped with bugs

New RP614v2 firmware resolved time-a.netgear.com by DNS instead of a fixed IP. But it accepted any UDP packet arriving on port 23457 as a valid reply, even from the wrong address.

Endgame A: serve them all

Option one was to embrace it. Put redundant NTP servers at three WiscNet sites sharing one anycast address, so requests land at whichever server is nearest and healthiest.

Endgame B: burn the addresses

Option two was to make the destination unreachable, which meant pulling a /20 out of global BGP. That sacrifices up to 4,096 university IP addresses for as long as the routers live.

The zombies never stop

Even while the server answered every request, hundreds of routers kept flooding because replies never reached them. Firmware in living rooms cannot be recalled, so the load outlives the fix.

“The unexpected behavior of these products presents a significant operational problem for UW-Madison for years to come.”