How to synchronize shared memory access

A forum for documenting the Parallella and Epiphany . (Not Q&A!)

How to synchronize shared memory access

Postby aolofsson » Thu Mar 06, 2014 12:19 pm

To simplify the hardware and to improve the scalability of our shared distributed memory architecture, we chose to employ a very loose memory ordering model. For exact details, please refer to chapter 4 of the Epiphany Architecture Reference Manual.http://adapteva.com/docs/epiphany_arch_ref.pdf

The most problematic memory sequence to date for programmers of the Epiphany has been Read after Write "RAW". Since the Epiphany employs independent read and write mesh networks it's theoretically (and practically) possible for a read transaction to overtake a write transaction in certain situations. It's very unusual for a shared memory architecture to not obey RAW ordering so this has given more than one seasoned programmer seriuous headaches. :D

The following code snippet shows how it's possible to guarantee completion of all read and write transactions to one particular Epiphany core. This example uses a dummy address in the remote core to guarantee that all outstanding read and write transactions have completed.

Code: Select all
  //Read data from sync location (READ)
  e_read(&dev, 0, 0, 0x6000, &sync_data, sizeof(sync_data));

  //Invert data
  sync_data=sync_data ^ 0xFFFFFF;

  //Write inverted data back to location (WRITE)
  e_write(&dev, 0, 0, 0x6000, &sync_data, sizeof(sync_data));
 
  while(1){
   //Poll (read) data to make sure new write data arrived   (READ)
    e_read(&dev, 0, 0, 0x6000, &poll_data, sizeof(sync_data));
    if(sync_data==poll_data){
      break;
    }
  }


The complete example can be found at:
https://github.com/adapteva/epiphany-ex ... e-mem-sync

It is likely that there are more efficient ways to synchronization memory. If you know of one, please reply to this topic or create your own working example. Note that there are many ways to skin this cat and this is not the optimal method from a performance method. The Epiphany architecture is a "write optimized" architecture, so if your are so inclined using a programming method whereby the receiver sends an acknowledge signal back to the transmitter is probably going to be more efficient...

Andreas
User avatar
aolofsson
 
Posts: 1005
Joined: Tue Dec 11, 2012 6:59 pm
Location: Lexington, Massachusetts,USA

Re: How to synchronize shared memory access

Postby timpart » Fri Mar 07, 2014 8:26 am

Andreas says the assumption at the start of this post is wrong, so this approach does not work, please ignore.
Does the TESTSET instruction go out to the target core via the write mesh? (I assume it does unless read mesh is 64 bit data path solely to accomodate this instruction) If so there is a possible alternative to the Read after Write sync problem.

Advantages: Core B RAM content effectively unchanged by the synchronisation. No flooding the mesh with polling read operations.

Comments welcome, especially spotting any flaws!

; Syncing remote Reads after Writes

; Program running on Core A
; Do your writes to Core B here

MOV R1,#%low(CoreBaddress) ; CoreBaddress can be any writable location in ordinary RAM
MOVT R1,#%high(CoreBaddress)
MOV R0,#0
TESTSET R0,[R1,R0] ; Do a test set on location CoreBaddress
; First a write goes from Core A to Core B (from way TESTSET works)
; This write arrives at Core B after any previous writes there from Core A program because of Write after Write guarantee
; The TESTSET acts at Core B. What happens here is cunning but irrelevant to the Read after Write sync
; if the target address is non-zero, it is unchanged.
; If it is target address is zero it is overwritten with zero in an atomic operation, effectively leaving it unchanged
; The outcome of the TESTSET goes from Core B to Core A via the write mesh
ADD R0,R0,R0 ; arbitrary instruction referencing R0, to force Core A to wait for the data to return from Core B

; Do your reads from Core B here
; Results from the reads are returned via the write mesh from Core B to Core A.
; They must come after the write the TESTSET did from B to A because of the Write after Write guarantee

Tim
Last edited by timpart on Fri Mar 07, 2014 7:49 pm, edited 1 time in total.
timpart
 
Posts: 302
Joined: Mon Dec 17, 2012 3:25 am
Location: UK

Re: How to synchronize shared memory access

Postby aolofsson » Fri Mar 07, 2014 11:06 am

Unfortunately the TESTSET goes out over the rMesh network :(
Andreas
User avatar
aolofsson
 
Posts: 1005
Joined: Tue Dec 11, 2012 6:59 pm
Location: Lexington, Massachusetts,USA

Re: How to synchronize shared memory access

Postby msegado » Fri Nov 07, 2014 4:50 am

Hi Andreas,

So, I understand that reads can overtake writes, and this is documented in Table 1 of the Epiphany architecture reference; however, doesn't the following statement from the same document seem to contradict that?
Load operations using data previously written use the updated values

Not sure what to make of this...

Best!
Martin
msegado
 
Posts: 3
Joined: Tue Feb 04, 2014 11:41 pm

Re: How to synchronize shared memory access

Postby Urhixidur » Wed Feb 18, 2015 8:31 pm

The e-mem-sync example on the git repository seems incomplete. The test.sh passes arguments (0 0 4 4 0x12345678) but the code (e-mem-sync.c) ignores any arguments (and has no usage() routine). The code allocates emem but does nothing with it. There are also several unused variables being declared (row, col, i, j).

Was e-mem-sync supposed to "test" each core?
User avatar
Urhixidur
 
Posts: 16
Joined: Mon Jan 19, 2015 7:19 pm
Location: Québec, QC, Canada

Re: How to synchronize shared memory access

Postby Urhixidur » Wed Feb 18, 2015 10:27 pm

msegado wrote:So, I understand that reads can overtake writes, and this is documented in Table 1 of the Epiphany architecture reference; however, doesn't the following statement from the same document seem to contradict that?
Load operations using data previously written use the updated values

Not sure what to make of this...

I think I've figured this one out. The full text is:
The only guarantees [of the weak memory-order model] are:
* Load operations complete before the returned data is used by a subsequent instruction.
* Load operations using data previously written use the updated values.
* Store operations eventually propagate to their ultimate destination.

I think they're using "written" to mean a store operation that has reached its destination. In this light, a memory location that is the object of a store operation is in the following successive states:
* value is up to date
(store operation leaves a core for the memory location)
* value is stale
(store operation reaches its destination and completes)
* value is written

A read (a.k.a. load or fetch) can get a stale value if it reaches the memory location before a concurrent store operation (a.k.a. write) reaches it (i.e. before it is "written").
User avatar
Urhixidur
 
Posts: 16
Joined: Mon Jan 19, 2015 7:19 pm
Location: Québec, QC, Canada

Re: How to synchronize shared memory access

Postby eepp » Tue Jul 21, 2015 5:54 pm

For those interested, this subject (wrong read-after-writes with the Epiphany) is discussed in a practical case study in this blog post (search for The Curious Case of the Epiphany).
eepp
 
Posts: 2
Joined: Tue Jul 21, 2015 5:41 pm


Return to Quick Start Guides , Documentation, Examples (Start Here!)

Who is online

Users browsing this forum: No registered users and 1 guest

cron