increase shared memory

Discussion about Parallella (and Epiphany) Software Development

Moderators: amylaar, jeremybennett, simoncook

Re: increase shared memory

Postby 9600 » Mon May 19, 2014 12:03 pm

Gravis wrote:change the Linux configuration then compile it.

file in the kernel: arch/arm/boot/dts/parallella.dts

change "0x3e000000" to whatever size you want then recompile the kernel. 0x3e000000 is 1024 - 32 megabytes.

Code: Select all
        ps7_ddr_0: memory@0 {
                device_type = "memory";
                reg = <0x000000000 0x3e000000>;
        } ;
.


The device tree source is compiled to it's own binary format using the Device Tree Compiler, dtc. The resulting devicetree.dtb file is stored in the BOOT partition of the Micro SD card. As such I shouldn't think you would need to rebuild the kernel.

Cheers,

Andrew
Andrew Back (a.k.a. 9600 / carrierdetect)
User avatar
9600
 
Posts: 997
Joined: Mon Dec 17, 2012 3:25 am

Re: increase shared memory

Postby schmurfy » Mon May 19, 2014 2:03 pm

I am currently experimenting and one thing I am trying to get running is using the epiphany to generate hash of photos to (later) find duplicates, that's what got me started in that direction but for now I am still trying to get a hashing application small enough to fit in the 32K of a core.
Once I have this running the other question is what should be done by the "main" system and what can be delegated to the epiphany cores, I was hopping to use the main linux system to just decompress the images and send them to the epiphany but an uncompressed image is quite heavy nowadays :)

My current idea is to scale down the image before sending it the the epiphany since that's what the hashing algorithm will start by doing anyway but 32MB still looks small. I get that the external memory will be slower than the internal 32KB but in this case I doubt I can get much into this 32KB especially after the program is loaded on the core.
schmurfy
 
Posts: 12
Joined: Mon Dec 17, 2012 3:22 am

Re: increase shared memory

Postby schmurfy » Mon May 19, 2014 2:07 pm

9600 wrote:
Gravis wrote:change the Linux configuration then compile it.

file in the kernel: arch/arm/boot/dts/parallella.dts

change "0x3e000000" to whatever size you want then recompile the kernel. 0x3e000000 is 1024 - 32 megabytes.

Code: Select all
        ps7_ddr_0: memory@0 {
                device_type = "memory";
                reg = <0x000000000 0x3e000000>;
        } ;
.


The device tree source is compiled to it's own binary format using the Device Tree Compiler, dtc. The resulting devicetree.dtb file is stored in the BOOT partition of the Micro SD card. As such I shouldn't think you would need to rebuild the kernel.

Cheers,

Andrew


Thanks for the precision, If I try this I would rather avoid rebuilding a kernel especially since I currently use the parallella itself to build my sources.
schmurfy
 
Posts: 12
Joined: Mon Dec 17, 2012 3:22 am

Re: increase shared memory

Postby timpart » Mon May 19, 2014 5:42 pm

shodruk wrote:
greytery wrote:If an E16 is 4x4 and an E64 is 8x8, and if only east addressing works on the Parallella, then we're looking at access to 2x52=104MB external memory for Epiphany IV / Parallela?


Unfortunately, no.
I think it gets worse on E64.


I agree with you shodruk. The only way to get the 100Mb+ would be to tie the column wires to ground so that the chip moves west to column zero, freeing up more room to the east. That would need say a tiny daughter board to plug into PEC_Power. (Don't make the row zero as well as that would put a core at (0,0) which is bad news.)

I also agree it is worse on the E64 as the cores go further east.

The only bit of good news is that the e-mesh goes east-west first then north-south, so you can have separate ranges available at different North-South positions. Each of the 16 cores could have its own 32Mb (or a bit more) contiguous range to play with. That would be 512Mb leaving 512Mb for the host to run in.

shodruk wrote:Or, can we manually configure the page table so that we can get contiguous shared memory?

From the Epiphany side, it is still noncontiguous, but it makes the data handling a bit easier.

I imagine you can do it somehow but I'm no expert in these matters.

Tim
timpart
 
Posts: 302
Joined: Mon Dec 17, 2012 3:25 am
Location: UK

Re: increase shared memory

Postby greytery » Mon May 19, 2014 8:24 pm

shodruk & timpart wrote:I think it gets worse on E64.

..it is worse on the E64 as the cores go further east.


No rush lads, but a quick explain :?:

tery
tery
User avatar
greytery
 
Posts: 205
Joined: Sat Dec 07, 2013 12:19 pm
Location: ^Wycombe, UK

Re: increase shared memory

Postby shodruk » Tue May 20, 2014 8:12 am

Tim,

I don't completely understand your idea, but that is interesting.
Maybe Adapteva knows whether that is possible.

tery,

the Core-ID of E16 is set to (32-35, 8-11),
so the E16 can only access its east address (0-63, 12-63).
e.g.
0x00c00000 - 0x03ffffff (52MB)
0x04c00000 - 0x07ffffff (52MB)
...

Maybe the Core-ID of E64 is set to (32-39, 8-15),
so the E64 can only access its east address (0-63, 16-63).
e.g.
0x01000000 - 0x03ffffff (48MB)
0x05000000 - 0x07ffffff (48MB)
...

I don't know why the origin of the Core-ID is set to (32, 8).
Shodruky
shodruk
 
Posts: 464
Joined: Mon Apr 08, 2013 7:03 pm

Re: increase shared memory

Postby timpart » Tue May 20, 2014 4:50 pm

ASCII art diagram

Here are the cores of the E16 and the positioning of the external DRAM

Code: Select all
E16

 -------   -------   -------   -------
| 0x808 | | 0x809 | | 0x80A | | 0x80B |
|(32,8) | |(32,9) | |(32,10)| |(32,11)|
 -------   -------   -------   -------

 -------   -------   -------   -------
| 0x848 | | 0x849 | | 0x84A | | 0x84B |
|(33,8) | |(33,9) | |(33,10)| |(33,11)|
 -------   -------   -------   -------

 -------   -------   -------   -------
| 0x888 | | 0x889 | | 0x88A | | 0x88B |
|(34,8) | |(34,9) | |(34,10)| |(34,11)|
 -------   -------   -------   -------

 -------   -------   -------   -------                                                     ---------------------------
| 0x8C8 | | 0x8C9 | | 0x8CA | | 0x8CB |                                                   | 0x8E0  to  0x8FF  32Mb
|(35,8) | |(35,9) | |(35,10)| |(35,11)|                                                   |(35,32) to (35,63) DRAM
 -------   -------   -------   -------                                                     ---------------------------


The same for the E64

Code: Select all
 -------   -------   -------   -------   -------   -------   -------   -------
| 0x808 | | 0x809 | | 0x80A | | 0x80B | | 0x80C | | 0x80D | | 0x80E | | 0x80F |
|(32,8) | |(32,9) | |(32,10)| |(32,11)| |(32,12)| |(32,13)| |(32,14)| |(32,15)|
 -------   -------   -------   -------   -------   -------   -------   -------

 -------   -------   -------   -------   -------   -------   -------   -------
| 0x848 | | 0x849 | | 0x84A | | 0x84B | | 0x84C | | 0x84D | | 0x84E | | 0x84F |
|(33,8) | |(33,9) | |(33,10)| |(33,11)| |(33,12)| |(33,13)| |(33,14)| |(33,15)|
 -------   -------   -------   -------   -------   -------   -------   -------

 -------   -------   -------   -------   -------   -------   -------   -------
| 0x888 | | 0x889 | | 0x88A | | 0x88B | | 0x88C | | 0x88D | | 0x88E | | 0x88F |
|(34,8) | |(34,9) | |(34,10)| |(34,11)| |(34,12)| |(34,13)| |(34,14)| |(34,15)|
 -------   -------   -------   -------   -------   -------   -------   -------

 -------   -------   -------   -------   -------   -------   -------   -------             ---------------------------
| 0x8C8 | | 0x8C9 | | 0x8CA | | 0x8CB | | 0x8CC | | 0x8CD | | 0x8CE | | 0x8CF |           | 0x8E0  to  0x8FF  32Mb
|(35,8) | |(35,9) | |(35,10)| |(35,11)| |(35,12)| |(35,13)| |(35,14)| |(35,15)|           |(35,32) to (35,63) DRAM
 -------   -------   -------   -------   -------   -------   -------   -------             ---------------------------

 -------   -------   -------   -------   -------   -------   -------   -------
| 0x908 | | 0x909 | | 0x90A | | 0x90B | | 0x90C | | 0x90D | | 0x90E | | 0x90F |
|(36,8) | |(36,9) | |(36,10)| |(36,11)| |(36,12)| |(36,13)| |(36,14)| |(36,15)|
 -------   -------   -------   -------   -------   -------   -------   -------

 -------   -------   -------   -------   -------   -------   -------   -------
| 0x948 | | 0x949 | | 0x94A | | 0x94B | | 0x94C | | 0x94D | | 0x94E | | 0x94F |
|(37,8) | |(37,9) | |(37,10)| |(37,11)| |(37,12)| |(37,13)| |(37,14)| |(37,15)|
 -------   -------   -------   -------   -------   -------   -------   -------

 -------   -------   -------   -------   -------   -------   -------   -------
| 0x988 | | 0x989 | | 0x98A | | 0x98B | | 0x98C | | 0x98D | | 0x98E | | 0x98F |
|(38,8) | |(38,9) | |(38,10)| |(38,11)| |(38,12)| |(38,13)| |(38,14)| |(38,15)|
 -------   -------   -------   -------   -------   -------   -------   -------

 -------   -------   -------   -------   -------   -------   -------   -------
| 0x9C8 | | 0x9C9 | | 0x9CA | | 0x9CB | | 0x9CC | | 0x9CD | | 0x9CE | | 0x9CF |
|(39,8) | |(39,9) | |(39,10)| |(39,11)| |(39,12)| |(39,13)| |(39,14)| |(39,15)|
 -------   -------   -------   -------   -------   -------   -------   -------

There is currently some spare address space between the East edge of both of the cores and the start address of the external DRAM. If we make the start address lower we can go as low as 0x8CC on the E16 (52MB) but only as low as 0x8D0 on the E64 (48MB) as there are cores positioned at addresses 0x8CC to 0x8CF.

So that is the limit for a contiguous range on the current set up. There is nothing to stop you creating additional contiguous ranges on other rows (including ones with no cores on).

With a daughter board or cable it would be possible to change the start column of the chip from 8 to zero shifting the chip left and making some more contiguous range available.

Tim
timpart
 
Posts: 302
Joined: Mon Dec 17, 2012 3:25 am
Location: UK

Re: increase shared memory

Postby greytery » Wed May 21, 2014 1:16 am

Thanks shodruck. Still struggling with the addressing concepts here .. - but having a lot of fun!

Tim - 'addressing memory on non-existent rows' .. Scary!
But what exactly do you mean?
And instead of moving to the left, what if we move to the right?

I'm not a HW/chip engineer- old ex-mainframe OS kernel hacker.
I don't understand how the FPGA is configured to map the E16 East-wards memory requests to the external DRAM, but it clearly does. There's some MMU magic/logic at the Zynq's Bank 35 interface to the E16.
OTOH, the FPGA also maps the ARM's memory requests onto the same DRAM - but represents that same physical space as different addresses for the host.
Seems that the memory can be as contiguous - or not - as the FPGA logic allows, or makes it appear to be. It's acting as a good old virtual paging kernel - like we had back in the good old '70s.

The FPGA provides the East eLink interface, then it must be aware of the Epiphany network routing protocol rules. For the default E16 placement of (32,8), then the FPGA may say : if the column number of an address request is less than 7, ship further East, else check row, etc. That gives a finite set of East and then North/South addresses. That is, along rows and then up/down columns that may have no physical cores on.
The further West/right we move the E16, there is larger FPGA virtual mapping space to the East because there are more columns. The potential for addresses sent to the FPGA interface by the E16 is limited to 4 rows (or they wouldn't get routed to the East interface), but the columns can be from 0 to 59. Similarly for the E64, but with 8 rows and 55 columns.

What would the addressing opportunities be if the E16 had coordinates of say, (32,60) to (35,63)?
(or some other position along the Western edge?)

Can the FPGA memory mapping really be this flexible?

The Parallella does not make use of the addressing to the West (a design choice somewhere), but that's effectively empty space.

tery
tery
User avatar
greytery
 
Posts: 205
Joined: Sat Dec 07, 2013 12:19 pm
Location: ^Wycombe, UK

Re: increase shared memory

Postby aolofsson » Wed May 21, 2014 6:41 am

The "magical" address translation is contained in these two files. 3 lines of code in total:-)

https://github.com/parallella/parallell ... rallella.v
https://github.com/parallella/parallell ... onstants.v

Code: Select all
assign ext_mem_access = (elink_dstaddr_tmp[31:28] == `VIRT_EXT_MEM) & ~(elink_dstaddr_tmp[31:20] == `AXI_COORD);
assign elink_dstaddr_inb[31:28] = ext_mem_access ? `PHYS_EXT_MEM : elink_dstaddr_tmp[31:28];


Andreas
User avatar
aolofsson
 
Posts: 1005
Joined: Tue Dec 11, 2012 6:59 pm
Location: Lexington, Massachusetts,USA

Re: increase shared memory

Postby shodruk » Wed May 21, 2014 9:06 am

Whoa! That's really magical!!!! :o :o :o :o
How much resource is needed to implement memory hole remapper?
Shodruky
shodruk
 
Posts: 464
Joined: Mon Apr 08, 2013 7:03 pm

PreviousNext

Return to Programming Q & A

Who is online

Users browsing this forum: No registered users and 3 guests