Bootable CPU-lane SSD on z390 quest

I am obsessed with performance. Any bottleneck I encounter is a personal insult. CPU performance gains are very minimal today, GPUs are still growing well, I/O advances quite well too. This one is about I/O.

So I have indulged myself with creating Raid0 out of 2x 970 Evo Plus on DMI bus. It works but not as fast as expected. Windows boot time seemed slow, while other tasks seemed OK. Except I knew it could be faster with my Asus ROG Maximus XI extreme DIMM.2 module. Maybe. So I tried to make it faster. I started with the following benchmark results

Low RND4K (normal for R0) and less-than-expected SEQ1M which should be over 6000MB/s, but bottlenecked by DMI

The numbers are high but not as high as I wanted them to be.

  1. There no any RND4k gain from RAID 0 compared to a single 970 EVO PLUS
  2. Entire raid performance is limited by DMI 4000MB/s maximum speed.

Attempt 1. Moving OS from RAID 0 on DMI to DIMM.2 module.

This setup is expected to connect both existing 970 EVO+ via DIMM2 which has dedicated PCIx4 lanes on CPU. It was supposed to avoid DMI bottleneck for RAID performance.

Starting Configuration
	|DMI bus
	  |-Samsung 970 EVO Plus
	  |-Samsung 970 EVO Plus
	|PCI Bus (Empty)

Configuration to be
	|DMI bus (Empty)
	|PCI Bus
	  |-Samsung 970 EVO Plus
	  |-Samsung 970 EVO Plus

To achieve this i had to

  1. Physically remove the SDDs from under GPU and heatsink. GPU removal and heatsink removal was required
  2. Attach the SSDs to DIMM.2,
  3. Create RAID 0 from scratch while losing all data on old RAID 0. I had a backup of course
  4. Rollout backup to the new RAID
  5. Find out 970 EVO+ raid on DIMM.2 is not bootable
  6. undo all the steps

Later research revealed that only Intel SSDs can be bootable on PCI lanes. possibly even in RAID config, but this is still unclear to me. The only intel SSDs faster than 970 EVO+ are 900 series. The only 900 series with M.2 slot is 905p 380 GB. One piece was 500 USD which is expensive enough. I decided not to go for RAID but to to try seemingly exceptional random I/O performance feel of the 905p with a single drive.

Attempt 2. Going Optane

So Intel SSD it is. 500$ later, I could finally install new 905p 380 GB into my DIMM.2 slot. The layout plan looked like this:

Starting configuration
	|DMI bus
	  |-Samsung 970 EVO Plus
	  |-Samsung 970 EVO Plus
	|PCI Bus
	 |- GPU (16x lanes)

Configuration to be
	|DMI bus
	 |-RAID 0
	  |-Samsung 970 EVO Plus
	  |-Samsung 970 EVO Plus
	|PCI Bus 
	 |- GPU (8x lanes)
	 |- DIMM.2 (4x lanes)
	  |- Intel 905p (SYSTEM, BOOTABLE)

This time I didn’t need to move hardware around the motherboard. The installation required mounting SSD to DIMM.2 and attaching it to the motherboard. Simple. And a little BIOS adjustment to reroute CPU lines from GPU to DIMM.2.

Software migration part was harder. I have copied my OS disk image with Acronis from RAID0 to the 905p. It didn’t boot at first, but booted after a couple of attempts, possibly installed some driver. To my unpleasant surprise, the 905p didn’t perform as per datasheet specs.

SEQ1M is expected to be 2500-ish and RND4K – around 250-ish

It was performing a lot slower! At this point i had no idea whether it was hardware or software issue. The device worked, but not as was as it was supposed to. I suspected it could be

  1. Wrong BIOS config
  2. Wrong driver/OS setting
  3. Wrong SSD place on motherboard (DIMM.2 instead of DMI slots) (OK, can put single 905p to DMI, it shouldn’t throttle the performance)
  4. Some exotic IRST Bottleneck, like inability to work on CPU lanes with z390 chipset or speed limitation when you still have your RAID0 running on DMI (PLEASE NOT THIS)
  5. Performance degrading after spectrum/meltdown hotfixes (NONONO)

BIOS config was pretty simple to check and rule out. in ROG motherboard the best you can do is to to designate 4 or 8 PCI lanes to DIMM.2. no other options. I tried both, got no improvement and moved on to Windows mysteries.

The only BIOS setting for DIMM.2 PCI lanes. this setting designates 4x lanes

Since the performance wasn’t as per specs it seemed bottlenecked by something. I was really hard to tell by what exactly. Apart of tests, the only clue I had was IRST suite PCI speed. It showed PCIe link speed of 2000MB/s while it was expected to be 4000MB/s. I didn’t make a screenshot back then, but I have a screenshot only now with correct settings.

Note the yellow highlight. It currently shows correct setting. When my Intel 905p was throttling, IRST was showing PCI link speed 2000 MB/s. this picture shows how your M.2 drive should look like

Finding the bottleneck and eliminating it wasn’t possible to me. I tried shuffling drivers around with no progress. Tried booting from fedora to check the performance out of windows with zero progress. I had limited options with driver shuffling since 905p was my OS disk. So I decided to reinstall windows from scratch. And it worked. Sadly, I couldn’t identify the root cause but now i know it was something related with moving OS sector-by-sector from RAID volume to single-drive volume which caused some unknown malfunction within windows. I’m happy it was assumption 2, not 3, 4 or 5. in this situation it would turn out to be close to impossible to fix.

Conclusion: you CAN have bootable SSD on CPU lanes with ASUS ROG Maximus XI Extreme with z390 chipset. Also, it’s likely supports RAID 0 on DIMM.2 on PCI lanes as when you are creating RAID in BIOS it allows you to add DIMM.2 drive to the drive pool. Maybe it can be made bootable RAID also, but hard to tell for sure. Motherboard documentation is really bad when it comes to describing this functionality. Same applies to Intel compatibility charts and any other datasheet i found. I know for sure it can be bootable. I know it can be part of RAID. Can it be both? go figure. I want to know but I don’t feel like spending extra 500$ for testing this.

Final BEST result: look at this RND4k!
Check out system responsiveness with it. Opening excel in milliseconds

MAB plugin for Windows NPS

My colleague has developed a MAB solution for Windows NPS.

Solution home post.

О даунтаймах

Блог лежал с 3 по 17 октября, потому что 3 отктября у меня истек домен и я пропустил нотификацию от регистратора. О проблеме я узнал 13го и ошибочно идентифицировал ее, как неполадки с веб-сервером, о чем написал в поддержку хостеру. Хотсер ответил в тот же день, что ДНС-запись адрес смотрит не на их IP. Затем я 4 дня был поглщен делами и не занимался траблшутингом, пока наконец не решил проверить, куда же смотрит моя днс-запись. Оказывается, домен истек.


  1. Мониторить доступность блога. Стендалон – это ответственность.
  2. Добавить в календарь напоминалки на даты обновления сервисов.
  3. Делать бекапы

Сапожник, как всегда без сапог. На работе во весь рост внедряешь процессы, итил, сервис менеджмент, а собственный блог лежит две недели. Это делает очевидным то, что мне на него уже давно положить. И снова посещает мысль, что надо больше писать.

Sendmail conspiracy

Sendmail conspiracy

Cisco IOS replace running config instead of merge

By default, if you do

# copy startup-config running-config

the startup config would be merged to the running one. The same happens if you use tftp:// or flash:// instead of startup-config. Eventually, there is a way to replace the running config. The command is:

# configure replace

Juniper Network Connect Eerror Fix

Diagnosis for me was getting error after a random amount of time running VPN to IVE.

According to Juniper forums and burgtrack [1, 2], The error message is caused by something changing your computer’s route table. It may be any application. hen it happens, you are going to receive similar message in a network connect log (when set to ‘Detailed info’ level: Advanced View -> Logs):

00218,09 2012/04/12 23:20:43.001 1 SYSTEM dsNcService.exe dsNcService p1612 t690 routemon.cpp:582 - 'rmon' Unauthorized new route to has been added (conflicts with our route to, disconnecting

The message is not going to be the last one before you receive the error, you have to scroll a bit up to find it.

Since I had no Bonjour installed, I had to find anything what could change my routing table. I searched registry for IP address appeared in logs. It should have pointed me to a software abusing my routing. In the registry of mine the address appeared few times, all in printing settings. Most of the branches having ‘’ record had ‘Hewlett Packard’ record, or appeared in HP branch, or had other relation to HP. Since the soft belonged to HP, I had to prevent anything developed by HP from starting up with my system. I used autoruns by Sysinternals to disable all the HP-vendored soft. After the reboot I discovered my VPN connection would not interrupt. After enabling services one by one, I found out the issue was caused by


I suppose it is some kind of driver or so. Most likely it is a part of printing driver ver for HP LaserJet M2727 MFP Series PCL 6 on Windows 7 64 bit. But I’m still not 100% sure that the issue cause by the driver itself. It may be caused by Windows 7, who attempts to add a route to the installed TCP/IP printer.

Perl error “isvstring is only available with the XS version of Scalar::Util” fix

I faced the subj error after updating Perl package. It appears, Scalar::Util package is shipped with C-compiled modules which are not properly updated by yum. The workaround is manual package reinstalling:

wget && \
tar zxvf Scalar-List-Utils-1.23.tar.gz && \
cd Scalar-List-Utils-1.23 && \
perl Makefile.PL && \
make test install && \

Bringing Cisco IOS CLI to Linux CLI

There are few people on the globe who loves to work with Cisco and Linux via CLI. These people might have issues with trying to apply Bash/Vim syntax to IOS and vice versa. I’m certainly one of them. That’s why I can do the followng in my Bash:

$ show .bashrc | i return
[[ "$-" != *i* ]] && return
#     return 0
#     [[ -z $adir ]] && return 1
#   [[ $? -ne 0 ]] && return 1
#     [[ $? -ne 0 ]] && return 0
#   return 0

It’s very handy for checking Cisco configs, stored on a Unix machines, without inverting your mind out. In fact, if you are in rush and tried to apply IOS syntax to Bash, you won’t be distracted by an error message, but you’d get a result you reqired.

$ show samle_conf.cfg | i spanning-tree
spanning-tree mode rapid-pvst
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
spanning-tree pathcost method long
 spanning-tree portfast
 spanning-tree portfast
 spanning-tree portfast
spanning-tree bpduguard enable

It’s achieved very easily. You need to add some aliases to your ~/.bashrc file and relogin:

echo 'alias show="cat"' >> ~/.bashrc
echo 'alias i="grep --color"' >> ~/.bashrc

Fixing SSH access on cisco via SNMP

Sometimes you may ecounter a situation, when your SSH is not properly configured, for example, if you forgot to generate SSL certificate before enabling transport input ssh on all vty lines, as I recently did. In this situation you might be lucky enough to have SNMP RW community string configured. In this situation you can fix literally everything.

There are no many configurable settings on cisco can be done via SNMP. But you can copy a prepared config to device via TFTP, RCP etc. You may download current device’s config to tftp server, edit necessary lines and upload it back. You may upload it to either running config, startup config or a flash file.

To download running config:

snmpset -v 1 -c rw_community hostname ccCopyProtocol.13 i 1 
snmpset -v 1 -c rw_community hostname ccCopySourceFileType.13 i 4 
snmpset -v 1 -c rw_community hostname ccCopyDestFileType.13 i 1 
snmpset -v 1 -c rw_community hostname ccCopyServerAddress.13 a tftp_serv_ip
snmpset -v 1 -c rw_community hostname ccCopyFileName.13 s "file_name" 
snmpset -v 1 -c rw_community hostname ccCopyEntryRowStatus.13 i 1

Edit on the server, and upload it back by the following commands. Be careful! If you upload to startup-config, IOS will not merge the uploaded config and the startup one, it will replace it instead. Do not upload partial sets of commands!. TO be on a safe side always I recommnd to never upload partial configs. Only necessary lines should be added/cancelled/corrected and the whole config should be uploaded.

snmpset -v 1 -c rw_community hostname ccCopyProtocol.13 i 1 
snmpset -v 1 -c rw_community hostname ccCopySourceFileType.13 i 1 
snmpset -v 1 -c rw_community hostname ccCopyDestFileType.13 i 4 
snmpset -v 1 -c rw_community hostname ccCopyServerAddress.13 a tftp_serv_ip
snmpset -v 1 -c rw_community hostname ccCopyFileName.13 s "file_name" 
snmpset -v 1 -c rw_community hostname ccCopyEntryRowStatus.13 i 1

If you ecountered situation with SSH with no generated certificate, You config might look like this:

line vty 0 4
 length 0
 transport input ssh
line vty 5 15
 transport input ssh

You should fix it to:

line vty 0 4
 length 0
 transport input telnet
line vty 5 15
 transport input telnet

Some commands can be cancelled with “no ” statment before the command. Some, as in above case, not.

Fedora LXDE UI Improvement Tips

Touchpad vertical scroll:

in file /usr/share/X11/xorg.conf.d/50-synaptics.conf

Section "InputClass"
Identifier "touchpad catchall"
Driver "synaptics"
MatchIsTouchpad "on"
MatchDevicePath "/dev/input/event*"
Option "VertEdgeScroll" "1"

Option “VertEdgeScroll” “1” – adds vertical scrolling

Restart X to take effect.

Adding keyboard layouts:

In file /etc/X11/xorg.conf.d/00-system-setup-keyboard.conf

Section "InputClass"
Identifier "system-setup-keyboard"
MatchIsKeyboard "on"
Option "XkbModel" "pc105+inet"
Option "XkbLayout" "us,ru"
Option "XkbOptions" "grp:caps_toggle,grp_led:num,terminate:ctrl_alt_bksp"

Define layouts to be used
Option “XkbLayout” “us,ru”

Define switching keys:
Option “XkbOptions” “grp:caps_toggle,grp_led:num,terminate:ctrl_alt_bksp”
grp – key to switch
grp_led – Keyboard LED to highlight a layout
terminate – shortct to kill X

Restart X to take effect.