Messaging Support

[timpart]

There seems to be a need by several people for some form of messaging service to pass messages between cores. So some kind of standard routine would be useful

I'm not convinced everyone wants the same thing though so perhaps people could explain what their requirements are.

I'll start with
Able to send message to a particular core
Able to pass message to own core
Able to send a message to a workgroup so that any member of the workgroup can pick it up

Message might be in local or external RAM.

Can send and forget (but perhaps not reuse the memory immediately)
Can send and wait for acknowledgement of completion of transmission (not the same as processed!)

Can retrieve a message sent for this core
Can retrieve a message sent for the workgroup

Any thoughts on typical size of message etc? Clearly external RAM has more space.
Can filter the above by message type (implies some type field present)

Technical
Fast
Supports heterogenous programs in the cores. So if you put messaging in your program it can communicate with a different program altogether in another core (May require changes to standard Linker scripts)
Easy to use (no complex set up in code)

Regards,

Tim

[aolofsson]

Tim,
Great inputs! We are working on it...
Andreas

[ysapir]

@timpart - thanks for your thoughts.

With the new eSDK5 we started implementing messaging infrastructure on the e-lib. The e_read() and e_write(), together with e_dma_copy() API's are basic primitives for implementing a messaging system.

Broadcast-to-group is a good idea and we will consider it for future release.

e_dma_wait() {which is embedded in e_dma_copy() as well} and e_write_ack() {planned for next release} provide the "message and wait for completion" functionality. Using e_dma_copy() is as fast as it can get on our architecture - but its interface was designed to be the same as memcpy(). So, a wrapper in the form of e_write() is required. For short messages, the setup time may be large relative to the communications time itself.

Implementation for heterogeneous programs is a little bit trickier and will require a messaging medium layer to accommodate the information on the physical location of the buffers.

[ysapir]

BTW, there's also OpenMP MPI for Epiphany in the works.

[mhonman]

This is probably heavily influenced by the needs of CFD, but IMO there are 3 basic kinds of message-passing needed and perhaps all of the use cases mentioned by Tim can be met by one of them.

They are all a kind of rendezvous, i.e. sender and receiver coordinate with each other to arrange the transfer of information. That means that code on each side that uses the message-passing abstraction does not need to be aware of the storage locations used by the peer process.

[list=][*]synchronous message-passing
Communication occurs when the two processes both explicitly request it. A process waits until its peer is ready, and both processes are blocked until the communication has been completed. [/*]
[*]gather-scatter operation
Used by a process which needs to exchange data with its neighbours. Takes a list of data blocks that must be sent to peers, and a list of buffers into which to receive data from peers. Transfers occur (pseudo) concurrently, call blocks until all are complete - each interaction with a peer is synchronous[/*]
[*]buffered gather-scatter operation
Data exchange, but buffered. Call blocks if a buffered exchange is already active, if not it sets up a work-list and returns immediately. Processing of the work-list is interrupt-driven and will need a scheduler to manage it. [/*]
[/list]

ysapir - are there performance advantages to e_dma_copy vs. e_write? The caller of e_dma_copy seems sit waiting for the transfer to complete, so is not able to do useful work while it is in progress... or am I missing something?

Mark

[ysapir]

[mhonman]

Thanks Yaniv, that's useful info - also your post on the .

So one more interesting to-do, namely work out at what message size it is worth changing over from programmed to DMA writes.

There might be a problem with using e-write and then sending an interrupt - the interrupt service routine is going to need to work out which location received data and/or which peer sent the message. Could there be a risk of the shared location being overwritten by another peer in between the end of the write operation and the sending of the interrupt?

If it is correct that master-slave DMA from other cores is interleaved on the receiver, that facility would be useful for passing control messages along the lines of 'Core (x,y) is ready to accept data in location Z' provided that the slave DMA engine will not accept more data until the DMA complete interrupt has been serviced.

BTW do we have any examples of master-slave DMA code that would assist with understanding its strengths and weaknesses?

[ysapir]

[mhonman]

[Gravis]