High speed network?

[vidarh]

This is "crazy idea" territory. At work we run our own private cloud. We're looking at transitioning to KVM using Sheepdog as the storage backend. Sheepdog is a massively parallel block storage. Think automatic RAID over the network etc.. Of course the downside is network bandwidth - for our current setup, gigabit ethernet is fine, but I keep eyeing up 10Gbps ethernet (or Infiniband...) but it's still ridiculously expensive, with port costs in the $100+ range.

Which is where the crazy idea bit comes in: The Epiphany chip has high speed links. Now, imagine if it can handle cable lengths sufficient for a number of servers in a rack. Imagine a 1U enclosure with a number of them in a mesh, and some links brought out to connect to the servers. Now imagine being able to run customised switching software + various offloading on the Epiphany chips. E.g. doing most of the TCP/IP and IP filtering before the host computers even have to look at the packets....

Even a full Parallella-on-PCIe, combined with a bunch of them connected together to make a switch might be cheaper per-port than a lot of 10GE switches, while a stripped down design ought to be able to beat the pants off them. Both price and feature wise. Assuming the links can handle decent cable runs (but for inter-rack links, even down to 1m-1.5m maximum cable length would still be interesting. Shorter would get dicy (but could still be doable it it'd be possible to bundle sufficient uplinks between multiple switches)

So, those of you who know the limitations of this thing better than me: Just how insane is this idea?

[shodruk]

Interesting!
How about an application server?
I think Epiphany should be able to accelerate well some sort of application server's query, like SSL, SQL, or PHP...

[vidarh]

[Gravis]

10Gbps ethernet is ASIC territory all the way. also, the problem with making a long link between chips is signal integrity. there is a reason ethernet cables only go so far before they need a relay.

[vidarh]

[hamster]

[Gravis]

[shodruk]

10 Gbps = 0.8M packets per second
(10 * 10^9 / 8 / 1500)

[timpart]

The eLink sends data a byte at a time down 8 wires plus a clock wire to say when to sample the data. The use of parallel wires brings its own challenges - if the signal path lengths differ too much the data won't get through. With a 1 GHz clock that is a nanosecond per cycle and an electrical signal goes about 20cm.

ELink sends a mode (1 byte?), 4 bytes of address then 1 to 8 bytes of data. If sending multiple lots of 8 bytes to consecutive addresses it just sends more data without the preamble.

The link is bidirectional so the 2 gigabytes per second refers to peak transmission rate of large amounts of data in both directions at same time.

There is also a wire to say stop sending to me, my buffer is full (more than one actually). The speed of signal transmission will limit the maximum wire length. Too long and the stop signal will arrive too late at the other end.

Tim

[vidarh]