Clustering and temperature variation

[optimaler]

@ Andreas: Thanks for that info. It's a little reassuring to know the variation in temps is somewhat expected.

@greytery: Inspecting the board I modded this morning suggests to me that the Artic Silver is okay for securing the heatsink, as long as the board remains upright. I suspect more of a tacky paste would be better though, at least for long term adhesion.

Also, I should mention, I haven't managed to disable the HDMI yet, so this is with HDMI going in the background (albeit not doing a whole lot). I'm not so savvy to some of the more device level manipulations (I'm a computational chemist/software engineer) so this is more or less with the default state of everything running. Also, bear in mind this data is not under load.

[9600]

@greytery @FHuettig I'll create a -utils repo and we can xtemp and other temperature monitoring and system admin etc. utils there. Once this contains a few useful tools I'd be happy to maintain a package to make it easier to bundle and upgrade them.

Cheers,

Andrew

[theover]

Probably it's best to enlarge the heat sink somehow, and make sure there's enough airflow. In an enclosure it's a long term thing to put air in an out actively, because the enclosure gets hot and the air in it too in the longer run (hours/day), but the board cooling is probably served best by assuring high air flow directly to the heat sink, and the larger the heat sink or the ribs on it in terms of surface area, the better the cooling works.

In PCs it's possible to draw away over 200 Watts from a I7 inch-by-inch cooler connections, and keep the enclosure cool enough with fans in and out over months continuous-on, so there's room to spare with just 10W! Maybe I'll have to look at a nice small "Sharkoon" brand fan, because they are really quiet and might make the small free enclosure work nicely instead of not being able to use the unit as a closed machine.

T.

[greytery]

@andreas - thanks for info on the 7020 chips. It's something to bear in mind but I don't think that variation takes it outside any limits, so nobody should expect a fried board.
I may not be the first user with temp details on the 7010, but it will be interesting to see how they pan out. Probably within the same distrubution as the 7020 though. I'll only have 2 x E16 to play with of course, and I may not have a E64 to play with for a long time....

@optimaler - sure the Artic silver stays gooey and sets after a while (trying to unhook a heatsink from an AMD or Intel CPU is always a delicate task!), but long term it's going to be a good idea to apply some pressure to fix it there. Even the 'tin foil' may have a tendancy to lose its grip occasionally, although it seems to work well enough for graphics memory, chipsets and such.
HMDI - I've still got to explore this definitively, but if there is no actual HMDI plugged in - as your cluster has it - I think that the circuitry is relatively dormant/cool. To be looked at... The attractive idea of reconfiguring the FPGA has come up, but that is way out of my league.

And, anyway folks, the board - as is - is still well within spec and should meet its primary purpose(s).
This is about post-sales-tweaking of the engineering.

tery

[theover]

The cooling is not needed to be better than inside the upper limit spec for most parts, but maybe these new chips have quamtum decay at some level, I don't know, that's rightly served by lower temperatures, an maybe the MOS chips are sensitive to moist still , and that that is less bad with lower temperatures. There are however parts that respond negative to higher temperatures: electrolytic capacitors, and I don't know if the number of decoupling capacitors probably on the various bif chips are free from this sensitivity. If there are relevant tantalum or elco parts (in SMD) on the little board, it could better stay in 50 degrees Celcius range if it's on often, to keep the lifetime of those parts good.

It makes the same sense in computer pcbs in general, keep them cool, and they last long, if not, they may well break, I know from practice. SO to keep a cluster working 24/7 for long, it might be advantageous to keep full power available for top speed on all parts while keeping the temp low, mainly this is easy by using a well-conducting bigger heatsink, that's probably easiest and most effective. I might even want to cool passive by simpling connecting a large enough heatsink.

Of course there's special thermal paste, which i've used to conduct a hundred or 250W away from I7s in optomal sense, but the size of and the airflow over a heatsink is more important, and 10 Watts or so per board isn't extreme, a decent moderate sized heatsink, one size class bigger than the super-tiny provided one should be ok. Howto put that on a cluster is another matter, maybe a big thick piece of aluminum per board connecting to a side-coolfin.

T.

[optimaler]

@theover: Your comments are insightful, so thank you for sharing. At a later date it will probably be worth investigating larger heatsinks, depending on how hot the boards run under load. Need to get the complete computing environment set up first. :/

[optimaler]

Okay, I'm beating a dead horse, but one last figure:

finaltemps.png (1017.77 KiB) Viewed 22414 times

This is after the application of Artic Silver to all my boards. The variation in temperature stayed consistent, so that lines up with expected leakages as Andreas has pointed out. So, if anyone has a board that runs hot, that's maybe expected (although bad for the lottery).

[greytery]

Actually, with all 8 in harness, it look's more like a team of Huskies.

Again, there doesn't seem to be any correlation between the position in the stack and the resulting temperature, but only further juggling and repeating the experiments will show. Probably time to move on and use your cluster for what it was intended.

The Artic results are consistent, and shows it's worth doing if you need to. I will, anyway.

I'll probably stop short of actual water cooling - but it can be done :

tery

[greytery]

FYI : An update for the temperature differences for 7010's can be found .
Seems we can expect similar variations between board samples, but the message is still: Keep Calm and Stay Cool!

Cheers,