Parallella Community

OK, given a set of processor cores each running APL, how do we communicate between them ?

(This is a continuation of the discussion begun in : Why port array languages such as Apl to Parallella ? )

A conventional APL system will offer a range of communication processes, now usually including object-oriented methods of accessing .net, Java, R, etc. For AplX see We need to find a set of communication methods that compactly gives us rapid communication between processes running on the cores of a Parallellla.

1) Well the methods that were built into aplc a number of years ago are based on piping :
a) using two pipes in for data and control, two pipes out for result and status : Basically based on std-in, std-out, std-err
b) piping data out of APL through a pipeline of shell commands and back into APL. A new mechanism []Pipe.
c) spawning a new long-lived process and connecting as many pipes to it as needed. A new mechanism []Spawn.
See the aplc.doc :


2) Another possible mechanism is a global APL variable handling program, that local APLs can make read and write request to. We wish to preserve the rigorous APL way of knowing the full specification of the data : its size, shape and type, and this information is kept by the global program.

3) Alternatively it is possible to just share a variable such that a number of processes share the data but the data is kept by the process with the most recent data used to write the variable. Any particular read then accesses the data (and its type and structure information) from this, so a data read is then a local to local process transfer over the global interconnect. This still needs to be 'policed' by a process with a global overview.

Techniques 2 and 3 are similar to the []SVC Shared Variable Control in AplX.

Your thoughts on these and other possible mechanisms are most welcome.
If you have the knowledge / ability / time to implement any of these using the local/shared memory system on the Parallella, that would be most welcome.

I expect you have been reading the Architecture manual already but here are some thoughts.

Reads are much slower than writes. A read goes on a slow link to the destination core then the data are returned by a write back to the originator. If you want to transfer data quickly have one core write it to another's memory, not by a core reading another's memory.

The DMA engines are good at transferring data around memory while minimising the impact of the CPU to do other things. (Best if DMA access is to 4K block of RAM not currently being used by CPU if possible.) There is a small overhead in setting up the DMA so it is not suited to tiny amounts of data.

Pipes naturally lend themselves to writes by the originator which is convenient. I'd outline a pipe mechanism as follows.

Each core that can be a destination for a pipe should maintain a list of available RAM areas in its local memory. It should use a mutex variable present in its local memory while maintaining the data structures describing the list. The mutex will be quick to access as it's local.

When a core wants to pipe to another core it acquires a mutex on the destination core's free area list and acquires a suitable region of RAM updates list, releases mutex.

Sending core sets up DMA to read data from its local memory and write to destination core's memory. DMA is started.

Sending core does other things. Destination core carries on obliviously.

A while later sending core checks DMA and discovers it is complete. Sending core notifies destination core that the data are available (by another mutex and work to do list probably)

Destination core looks for more work and notices sending core's notification in its work list.

Tim

Tim this is great.
Yes I had noticed that the DMA channels looked like important channels, but had got no further.
I will be brutally honest and say that I have hopes that someone or someones (as knowledgeable as yourself), will take it upon themselves to build pipe mechanisms. Whether tried and tested means, or more novel means particularly appropriate to these processors, will be the way forward, I do not know.
I will have quite a bit of my time taken in just getting the Aplc basically up and running to the best in such a shared memory context, where the size, shape and type of the data is preserved in the transfer. But pipe data can (initially anyway) be plain text streams of arbitrary length.
But certainly the first things that are needed are simple standard in and standard out pipe communication to the host processor.
I am trying to hack a simple fixed buffer substitute, but I am sure someone who reads this has better tools in their kit-box.
cheers,
Beau

I haven't had a chance to look real in depth yet on the Parallella, but are hardware mutexs available? A mutex per core would make any sort of inter-core communication much easier, as any internal or external resource could "lock" a core for use. I could imagine this leading to issues if a core got frozen, but have a watchdog per core that would auto reset could probably get around this problem.

It is also a question of how DMA or a process end on another core would grab the mutex if a different core already has the lock. If multiple cores are all vying for access to the same process, this could likely get very interesting, which shows the need for the high-speed FPGA to manage these things. Maybe support a table of mutexes in the FPGA?

I feel like I'm missing something that is probably in the hardware design already though.

Mutexes can be done by the TESTSET instruction on the Epiphany chip. The memory location must be on same chip as the core executing it. Can't be on external DRAM. Haven't seen anything to suggest the FPGA can do this to Epiphany internal RAM so would assume for time being not possible..

Tim

Yes @ysapir should have mentioned the mutex functions, sorry was in a hurry and wanted to answer "is there hardware support" and convey that my answer was about Epiphany, not ARM which can also do hardware mutex but differently. (Can't remember how right now - it's changed since I first came across the chip back in the days when the A stood for Acorn.)

Would be good if one day we could get a mutex across both chips somehow. At the moment needs careful keeping out of each others way in the shared external DRAM unless I've missed something.

Tim

Please look at this post: