ASUS WRX80E Sage SE WiFi II & Threadripper Pro 5955WX Review: PCIe Lane Heaven or Proxmox Nightmare?
next to me on the bench today is the Asus Pro WS wx8 Sage SE Wi-Fi 2 motherboard and this promises to be everything a pcie lane junkie such as myself could ever possibly want in a home lab server unfortunately though it's not as quite as straightforward as that and we'll get into that throughout the video the platform seems to be played with some weird little niggles little weirdnesses it's also getting a bit old I decided to pair this meiger B of a motherboard with the thread Ripper Pro 5955 WX CPU and 512 gigs of ddr4 ECC memory 2 this is a glorious Powerhouse a a symphony to home labbing it's got fast single-threaded performance fantastic multi-threaded workload performance too so I guess we'll just end the video there right not so fast because there are a couple of elephants in the room we should discuss before we get to them though namely the price tag and discuss why getting Pro MOX to even boot was such a pain in the dairy air let's take a look at the hardware itself because this combo is pretty special released in March 2022 the AMD ryzen thread repper Pro 5955 WX 16 core 32 thread zen3 chip sits at the entry level of amd's workstation focused thread Ripper Pro 5000 lineup I do find it funny when a CPU that costs $1,000 is referred to as entry level but when we remember that the 5,000 series of threader Pros goes all the way up to north of $66,000 yeah I can kind of understand where that comes from the 5000 series is based around the zen3 architecture fromd which is built around the 7 nanometer process spreading 64 megabytes of L3 cache over two ccds now that layout becomes important when we start talking about memory performance in just a second but the base clock of this chip is the fastest available in the thread repper Pro lineup at 4 GHz base clock turboing up to 4.5 GHz now for those of you looking to run a lot of virtual machines and do just a lot of parallel processing those fast single clock speeds become important for single-threaded tasks like wire guard encryption unzipping and decrypting files uh things like just Windows just like single thread speed to to run quickly uh even in a virtual machine so desite the fact that you've only got 16 cores available they are 4 GHz cores which should give you quite a lot of Headroom when it comes to running all the virtual machines that you could think of there are eight dim slots here configured in an OCTA Channel configuration so ostensibly what that means is you could put seven sticks of ram in this or five or three like you don't have to do pairs of sticks of ram like you do in consumer grade systems for example the downside of the Dual CCD layout though so AMD ryzen chips have a a chiplet layout so they basically bolt a couple of CPU dies together into what called smaller chiplets you have to have some kind of a connecting fabric between those dyes and the scheduler has to pick somewhere to put and allocate the RAM and the processes so if if you've got a bunch of stuff loaded into context on one CCD and then for some reason the system picks the other CCD to run the task there's going to be some extra latency in involved in that kind of transition from CCD 1 to two that generally hurts the maximum available memory bandwidth as discussed in that Reddit thread linked below good news on the pcie front though this is a Gen 4 capable system with a full 128 lanes for massive expansion capabilities you can bate pretty much all the slots individually although I did see a note in the manual for the motherboard stating that this board despite the fact that each slot is IND individually bable you can only bate up to 10 ssds in here so just you know be aware there are some weirdnesses even though you've got all the lanes available depending on your motherboard you might run into some weirdness and how you can actually take advantage of that now you can see here this is where the 5955 WX fits within the 5000 series lineup at the top we have the 5995 WX a $65,000 CPU which has quite a slow Basse clock but you more than makes up for that for multi-threaded workloads with the 64 cores and 128 threads available the trouble with this CPU at this point though is that the Zen 3 architecture is just getting a bit old it was released in March 2022 so 3 years ago as of recording and of course the newer epics and thread rippers are now beginning to become available 7,000 is available and I've read rumors that there will be a 9,000 thread Ripper coming too despite some people claiming that thread Ripper is end of life and those bigger chips have distinct advantages if you need them notably vastly more PCI Lanes than the Consumer grade counterparts but it's hard to ignore the sheer raw performance of something like the newer Zen 5 chips with the uh is it the 99 50x for example uh this is based around the Zen 5 architecture from AMD and you can see here in this chart that the the sheer performance the sheer grunt available from that 950x actually puts this zen3 part to shame despite the fact it uses less power despite the fact it's half the price the new architectures are just faster single core performance is better the 950x significantly outperforms the thread Ripper Pro in single core benchmarks with a 50% higher score in geekbench 6 this is largely due to higher clock speeds and also some architectural improvements too with multicore the same story is kind of true too where despite both CPUs having the same core count 16 cores and 32 threads the 950x leads by about 35 to 66% in multicore benchmarks across various tests this Advantage is likely due to Zen 5's improved efficiency higher IPC or instructions per clock and those faster clock speeds that we talked about now we come on to power efficiency and the story continues to be in the favor of the ryzen 9 9950 X which has a TDP of just 170 watts compared to the thread Ripper Pros Hefty 280 Watts this makes it far more power efficient which is reflected in its higher performance per watt metrics now this is probably really important for those of you wanting to do this in a home lab scenario whereas the thread repper Pro is kind of more aimed at that splitting that difference between a real server and a desktop like a gaming rig really um so that's what the workstation class boards are really designed for so the threader Pro is probably best for those users that need massive memory bandwidth ECC memory support and huge amounts of PCI lanes available to connect a metric boatload of PCI devices to their systems this makes it ideal for things like rendering Farms virtualization servers although we'll get on to that in just a minute and things like uh scientific rendering and you know data processing workloads that just need that sheer grunt of you know many cores and lots of power and lots of things and lots of stuff connected to a box the 950x though might be a superior choice for desktop users who prioritize raw CPU performance for things like gaming or generalized workloads where those higher clock speeds and the power efficiency are more critical than the memory bandwidth or the number of PCI lanes available now let's take a look at the Asus Pro WS WRX 8e Sage SE Wi-Fi 2 and the two moniker is important here this is because the s RX8 socket that the thread repper Pro sits in is uh uncapped they've they've made it so you can actually overclock this CPU on this motherboard the version one you couldn't but the version two you can now there are seven pcie 4.0 16x slots here Gen 4 slots the WRX 90 board the next chip up has Gen 5 slots but this one because it's a few years old they're Gen 4 slots as discussed there are eight Ram slots available in an to channel configuration eight8 Channel configuration and the storage options are pretty beefy too there are three m.2 pcie gen 4 slots eight sata 3 slots as well as a pair of UD do2 ports which can be configured in either PCI mode or nvme mode or SATA mode so you can get sff breakout cables that will connect each of these slots to four hard drives each networking is pretty good story too there is a pair of Intel x550 nck on the back using RJ45 connectors I think I would like to see in boards like this the option of an SFP plus connection I mean there's a pair of Nicks back there why can't we have one that's RJ45 and one that is SFP plus the RJ45 stuff just runs so hot and then I need I need another connector on the end to convert that back to RJ45 back to SFP Plus in the switch anyway it's a minor niggle but I would just love to see it there's also Wi-Fi 6E if you need Wi-Fi in a server now the other thing that I haven't mentioned yet is ipmi in here you can actually Connect into one of those Intel Nicks and get a full ipmi experience remote management experience built directly into the motherboard it's a pretty good implementation it's similar it looks a similar theme to The ASRock rack implementation on some of my other boards uh but you can control things like fan speeds look at sensors power the system on and off install an OS all that kind of stuff no extra licenses required no weirdness like idra or anything like that now when it comes to installing this Beast of a motherboard into a server I'd be remiss if I didn't mention that the board is physically huge it's big in practically every single Dimension now it states that it's an e8x board but in fact it's actually slightly wider than the e8x spec and that becomes a real issue when you start connecting things down here in the bottom the pcie power slots that are down there so I actually ended up having to use this breakout cable which is a single six pin to a pair of six pin PCI P this is from my old Bitcoin mining rig i' I've saved it for all of these years and you know everybody has a box of cables somewhere they're saving for for a rainy day one day well this was mine this was my win this week um but the trouble is there isn't enough clearance because the rather than the the connectors being upright like this they're angled at 90 degrees so if you picture this in a case you've got the power supply here and the sidew wall of the case right here certainly in my rose will lsv 4500 that I was using which is a standard 19in rack case and that's a big case that just is there's only fractionally enough room to squeeze those power connectors in between the side of the case and actually get them into the motherboard and in fact I had to lift the board up put the cables in and then kind of shove it down a bit to get it to fit and the same is true of all of the other connectors on the side here too there's no way you're getting the USB 3 you know this this style header for example there's no way you'd bend that if it was on the bottom here um down there as well you've got a a breakout for a VGA connection there is a no VGA port or no fact there's no display out on the back of this motherboard so if if you're not using the ipmi or a graphics card I think it's a fairly safe bet to assume that most people doing a workstation with this are using a graphics card but for those of us doing a headless server just a VGA port would have been nice I mean I I kind I kind of like I kind of understand why there is no VGA port because they're quite big and bulky and old at this point but also having a separate header sort of here and then dangling a cable underneath my gpus and what's already a pretty congested area anyway not the best choice in my opinion but uh the other thing to bear in mind when it comes to power is so not only do you have to connect up this pair of auxiliary six pin connectors here because there are so many PCI slots that it needs extra power because each slot can pull about 75 Watts I think per the ATX spec so 7 time 75 Watts that's a lot to go through here so not only do you have two 8 Pin connectors for the CPU the 24 pin motherboard power connector you've also got an extra 8 pin connector here because the CPU can pull up to 280 Watts you've then got a pair of six pins down here so by the time I'd hooked up my power supply to these two eights this eight and these two sixes plus my graphics card the Nvidia A4000 I was out of cables I was out of you know it's a brand new it's the GX 1200 it's a brand new power supply and I was out of cables so I ended up having to use this Bitcoin mining rig splitter as I talked about which takes a single six to two sixes shov that in the bottom and you know it was fine because I'm not completely maxing everything out in this board but just something else to be aware of that you're going to have to plan your power supply for this board pretty carefully now let's talk about power usage for a moment and let's just start start this section by tempering Expectations by saying that despite this CPU being the lowest core count in the thread Ripper Pro range and the cheapest it's a toasty boy full load no PCI devices with a stress test a synthetic CPU load was very easily able to get this to pull 360 Watts from the wall and idle was 180 Watts so it's a it's a pretty beefy system and by the time you've added a few pcie devices you're look look at a pretty easy system idle of 220 to 250 watts and then by the time you switch on a Windows VM or something like that in the system you're going to add another 100 watts to that just to idle so just plan your cooling accordingly because this is a toasty toasty boy so now we get into the proxmox booting problems that I had with this motherboard this this caused a few extra gray hairs to appear in my beard so a Long Afternoon turned into an evening of hacking when the vanilla proxmox ISO just would not boot it would hang after selecting boot at the loading initial Ram dis option so I ended up supplying a kernel command line argument of PCI equals nomc after Googling around for God knows how long and the installer finally booted so I was able to proceed as usual without incident however the system after install still needed PCA Nom comp to to actually even boot so I then looked at upgrading the BIOS and flashing The BMC which was done using the inbuilt ipmi and it was kind of painless really but uh that reset all of my bio settings I've been tweaking around with all afternoon too and so I ended up reaching out to Wendell who helped me figure out the esoteric combination of BIOS settings that were required notably IMU this was set to Auto in the bios for me but I had to actually physically go in and change that to to be enabled despite the fact that I think AMD CPUs haveu enabled out of the box by default I had to actually go and physically change that bios setting there was also another setting of pcie 10bit tag support that I had to update to as well as a few other things which you can see here in the screenshot too I'm not sure which one of these settings actually in the end made the difference allowed me to boot without the pcie command line flags of nomcom but once I had booted I started noticing AER errors or Advanced error reporting errors in D message there was a bunch of stuff going through here related to bad TLP and just a bunch of stuff related to storage and so I used a command called lspci DTV to determine which devices in the pcie tree were causing these issues and initially the A4000 was showing up along with a pair of tsheet data center grade nvme ssds in the onboard slots down here so at this point I jammed in my 2 pcie six pin orc power cable and the A4000 errors went away so just bear that in mind if you are seeing PCI errors those two ports down the bottom don't seem to be optional however despite that the Toshiba SSD related errors remained now some folks suggested setting the pcie link speed to gen 3 to try and resolve this issue but so far as I could tell there isn't a way to do that for the onboard slots now I could put the tashiba drives into one of these cards and then bate that and then set the link speed in the Bios that way but when it came to the onboard slots I couldn't figure out how to do that or indeed if it's even possible now are these errors harmless my research tells me maybe in fact if you look at the wording of the error message it says the hardware error has been corrected and no further action is required so in terms of whether there's aned imediate concern or not I think the corrected areas are fairly common in high performance systems at least according to my research anyway especially those with complex pcie topologies like the workstation board in front of me they don't necessarily indicate failing Hardware but what they can sometimes indicate is issues or minor issues with signal Integrity or some kind of a firmware bug and the frequency of these errors was what concerned me roughly every one to two minutes I was seeing the errors printed out and in in general the research that I did told me that if these errors were sporadic they can generally be safely ignored but I I just don't feel comfortable building a new server around such an expensive platform and seeing these errors come out in the D message logs every minute or two you can add PCI equals no aeer to your kernel command line arguments and I believe there used to be a bio setting at least Wendell said so uh but there there doesn't appear to be one anymore U and that no aeer setting will suppress the errors in the logs but they're still there underneath so yeah I just I just don't feel comfortable with it I also don't want to downplay the frustration I had with the process of just getting the proxmox install ISO to even boot I should say that NX OS booted straight away no issues Arch booted straight away no issues same with auntu but for some reason proxmox despite the fact it's using an auntu kernel it was just being it was just being a diva so I even tried upgrading my kernel from the default I think it's 6.8 that proxmox ships right now to 611 made no difference unfortunately but the long post times whether it's due to the amount of Ram or just a platform issue of you know one two three minutes each time make this a really long drawn out frustrating process where you just you sat there just tapping the delete key or the the select boot option key just to try and get into the bios so a big thanks must go out to wend for helping me figure out some of the bio settings for boot parameters even if they didn't ultimately end up helping me much with the bad TLP or bad dllp errors that I was experiencing and it seems to me that thread repper Pro is a bit of a picky platform when you look at the the general blogos sphere or forom sphere of posts that people make about this stuff there are lots of people having similarly esoteric issues as myself with this board maybe I was just spoiled by the experience of building in the Epic 7402 BAS system last year where everything just worked first try so rare but when it happened I mean in that video I remember being a little like it just worked like it was no big deal but this is a reminder to me of just how how rare and how nice that experience was so despite those negative things PCI pass through on this board was a piece of cake getting it actually working was pretty much a non-event it just worked the io mmu groupings were good like my graphics card had it own dedicated group in fact each slot seemed to have its own group and overall performance was fantastic you know i' I've grown fairly used to the speed of my servers over the last year or two and this thing is a notable step up so I can only imagine what it's going to feel like if I built a server around the 950x for example that we talked about earlier so what are the conclusions then well I think it's it's clear to me that this board has as my school teachers would have say it has a has a a potential if only he'd apply himself there are dozens of posts with this motherboard and the previous version trying ever more random fixes to try and resolve the aeer errors and let's not forget that this is still an expensive platform to buy into $22,000 for just a CPU and the motherboard coupled with the fact it's a three-year-old platform too I kind of expected better from a stability point of view like I expected the Linux Kel to have picked up all these little weirdnesses and the errors logged might not be terminal but they just they don't give me the warm and fuzzies That I Want from a system like this maybe the fact I had to spend three or four hours just to get an OS installed and reliably booting soured me on the entire system maybe it was the price maybe it was the heat that I could feel emanating from the back of the system at all times I sat next to it it was a combination of all three maybe it was just the fact that I saw AMD released the 950x 3D this week as well I think if I could find this platform for a good price let's say something around the price of building a 950x based consumer system uh it would make a lot more sense but as it stands this is a $2,000 platform and I could build literally two 99 which is a top-end desktop chip I could build two 950x based systems for the price of this one so paying full sticker price for something that feels I don't want to say Half Baked but just has little niggles and little issues because thread repper Pro isn't mainstream so it just doesn't get the same attention that epic does in the term in in in regards to the server platforms and the am5 the Zen 5 stuff for the 950x in the desktop side it just doesn't get the same attention from Asus from AMD as some of the other things in the lineup it fits in this weird kind of Middle Ground that nobody's paying much attention to it's also a bit power hungry honestly you know just that idle power being north of 200 Watts for me is a a bit too much and it's a shame because the performance of this chip is actually really pretty good for what I need it for and all of the PCI Lanes the bif foration stuff all of that all of that just combined to make this really a Bittersweet review for this motherboard I really wanted to like it but in the end the newer Zen 5 consumer chips are just so fast and relatively affordable compared with this system as I say I could literally build two for the price of one so with some careful planning upgrades and consolidation of devices I think for my use case at least I could probably make a pretty good case for a Zen 5based desktop chip for my future server although that might be a future video so for now I think my time with the thread Ripper Pro platform is about to come to an end so I might go and brave the thunderstorm that's raging outside and go and return this thing to Charlotte thank you so much for watching and until next time I've been Alex from ktz systems
2025-03-29 07:07
