Ethernet performance

[kunzej]

Hallo,
I'm a German physicist working for the NA62 experiment at CERN.

I'm responsible for the online PC-farm at NA62 which will be installed in Summer 2014 and needs to process more than 200 Gbps data. So far I planned to use about 60-100 PCs with one 10Gbe network device each. Now I'm thinking about using Parallella boards instead of the PCs. Because of the limited memory I would guess that we need at least 500 boards (16 cores should be enough).

So my question is, is there any information concerning the 1Gbe performance? What is the maximum rate you can read let's say 1k packets, how stable is it and how do you distribute the packets over the Epiphany cores? How big are the latencies?
It is very important that we are able to receive UDP packets without loosing any (as in less than 1 packet out of 10^8)! I know that this is not too easy at the full 1Gbps rate and I've spend a lot of time to manage to receive the full 10Gbps with a PC. So are there any performance tests? Probably this will be the bottleneck.

Apart from the performance: Will you be able to ship ~1000 Parallella boards until next Summer and is there a way to request some prototype boards soon?

More about my work: I'm in contact with two much bigger experiments at CERN, namely LHCb and ATLAS. Within their current upgrade they are heading similar problems as I do. LHCb is thinking about installing a huge ARM-based farm to replace and upgrade their current farm with ~2000 PCs. So I think there is a very big potential for Parallella boards at CERN.

With best regards,
Jonas

[aolofsson]

Hi Jonas,

Thanks for the post! The potential of having Parallella be part of CERN project in some way is very exciting! The board has a dual core A9 that runs Ubuntu so we can easily run some speed tests between boards if you let us know what you want to test. Is there a specific test code you want use to test? Distributing packets to the Epiphany cores is a little trickier. Right now there would need to be some kind of proxy running on the ARM that would parse the packets coming off the Ethernet port to send the data to the Epiphany cores using our light weight driver (write/read/open/close etc)

Andreas

[kunzej]

Hi Andreas,

thank you very much for your quick answer.

If I get you right there is a way to skip the A9 and directly access the packets from the Epiphany cores. In this case I don't see a reason to do any performance tests running on the A9 as the packet distribution is always the bottleneck.
What I want to test is quite simple: How many UDP packets of the size N can you receive and distribute to the remaining cores per second, without loosing any packets. If you can reach something like 0.3 Gbps with N~1kB and you still have enough space for the actual data processing I am happy.

The problem with PCs and using the Vanilla Kernel Sockets is that in principal you can receive even 10 Gbps but due to interrupts/context switches it sometimes happens that the kernel drops some packets (even if you only receive with a reduced rate like 1 Gbps). So our current framework is based on dma+polling which works very well.

What do you mean by "Distributing packets to the Epiphany cores is a little trickier". So I guess there is nothing implemented yet. Are there any plans to provide a framework for that soon? If not, I would love to contribute to that.

Jonas

[aolofsson]

Hi Jonas,

In the Parallella system, the Epiphany cores and the ARM A9 share the same memory space and both can act as a master. This means the Epiphany can do some nice things (and some not so nice things ) within the system independent of the ARM.

We haven't implemented anything that grab packet out of main memory and copies them to the the Epiphany cores so of course it would be awesome if you could contribute something in this area! Moving data between the zynq and the Epiphany is basically a memcpy. Alternatively, there is the potential of the DMA engine in the zynq could grab packet from DRAM and send them out to the AXI bus, where they would be routed to the Epiphany cores based on the memory mapped addresses. If the packet are really 1KB, they would fit very nicely within the epiphany local memory.

Andreas

[kunzej]

Hi Andreas,

can you please guide me through the specs I have to read to understand what needs to be done for your suggestions. Is there any HDL adaption needed for the DMA?

But before this: I really have to know if you will be able to ship so many (~1k) parallella boards until next summer and if there is a way for me to get one prototype within the next months. If not I'm still interested in your product, but I will have to change my approach.

Jonas

[aolofsson]

Hi Jonas,

Here is the link to all the reference documents for easy reference. I would start with the Epiphany architecture reference manual: http://www.adapteva.com/all-documents

We are scheduled to ship 6,300 boards already by the end of the summer and the plan is for there to have the capacity to produce 10K boards/month (more if other vendors pick up the open source design). This may sound like a lot, but these are very small boards that run through the machines at a high rate. We can get you a zedboard based prototype system immediately if you want to get going. Just send an email to info@adapteva.com.

Thank,
Andreas

[LamsonNguyen]

I approve of this thread.

[9600]