• Skip to main content
  • Skip to footer

NetworkJutsu

Networking & Security Services | San Francisco Bay Area

  • Blog
  • Services
  • Testimonials
  • About
    • About Us
    • Terms of Use
    • Privacy Policy
  • Contact Us

Home Lab

VMware ESXi Home Lab [2016]

03/19/2016 By Andrew Roderos Leave a Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

I recently bought an Intel NUC 6th generation (Skylake-based Core i3) as a VMware ESXi host. This is an addition to my existing host that I assembled back in 2012. It has served me well over the years and hope that it will continue to do so for several more years. Four years old in the computing world is obsolete, but I think it is still a very capable machine for what I use it for. Having said that, it will still be running as another ESXi host.

Related: VMware ESXi on 10th generation Intel NUC (Frost Canyon)

You might be asking why in the world would someone need two ESXi hosts at home that is not even studying for VMware certifications. I do, however, want to point out that I wanted to take the VCP certification back then that was why I attended VMware vSphere Install, Configure, and Manage [V5] class in a community college back in 2012 – check my tweet about it. Anyway, the primary reason is that the 32GB RAM on my first ESXi host is beginning to be too limiting for my CCIE lab purposes. I have to manage the amount of VMs that are turned on or I will be doing a memory overcommit.

The secondary reason, which is related to the first one, is that I really want to run vCenter Server so I could play with vSphere Flash Read Cache and other stuff. The Flash Read Cache feature was mentioned to me by @matthaedo on his response to my tweet. I did a quick search about this feature since I did not know what it was, how to do it, etc. and found out that one of the requirements to take advantage of the Flash Read Cache feature is vCenter Server, which I did not have. Yes, I could’ve installed the trial version of vCenter then test it out in my existing host and be done with it. But, I really wanted to add another host anyway so that it can handle my other VMs and add vCenter Server Appliance. Fortunately, I found out that we have VMAP (VMware Academic Program) campus subscription. One of the things that this subscription gives us is that it allows any student, staff, or faculty to use VMware infrastructure products for personal use to gain hands-on experience. The program is similar to the VMUG Advantage but only few products are included. For example, we do not have access to VMware Virtual SAN in the portal but the VMUG Advantage’s EVALExperience does. So, if one is looking for VMware licenses that are relatively inexpensive ($200) then the VMUG Advantage subscription is the way to go. I just wish it includes VMware NSX though.

Let’s go shopping

Intel NUC

At the time of writing, the Intel NUC 6th generation has four models shipping, two of them have identical Core i3 CPU and the other two have identical Core i5 CPU. The four models are NUC6i3SYK, NUC6i3SYH, NUC6i5SYK, and NUC6i5SYH. The main differences between identical CPU models are the size of the enclosure and the SATA3. Models with SYH at the end have the space for 2.5″ HDD or SSD and of course the SATA3 connector for it. The NUC6i7KYK – Skull Canyon is now available for purchase and ESXi 6.0 U2 can be installed with BIOS changes.

There have been several virtualization folks who had been running Intel NUC for years. The earlier NUCs are not compatible with the official ESXi ISO so it requires a customized ISO that contains proper drivers to install ESXi. With the 6th generation, it is no longer needed to create a custom ISO making it a seamless install.

Without further delay, here are the parts of my ESXi host:

1 x Intel NUC6i3SYH
1 x G.SKILL 32GB (2 x 16GB) F4-2133C15D-32GRS
1 x Sandisk Cruzer Blade 8GB USB Flash Drive (not pictured)
1 x OCZ Deneva 2 C Series 240GB D2CSTK251A20-0240 (not pictured)

I just happened to have the USB flash drive and OCZ SSD drive so feel free to buy the alternative parts listed below:

1 x Crucial 32GB (2 x 16GB) CT2K16G4SFD8213
1 x Sandisk Cruzer Fit 8GB
1 x Samsung 850 EVO 250GB 2.5″ SSD and/or 1 x Samsung 850 250GB EVO M.2

If you decide that you want additional NIC, then you may want to purchase the StarTech USB 3.0 NIC. Right out of the box, it won’t work but with a little tweaking then you could definitely make it work.

At this time of writing, the Crucial ($165) is cheaper than the G.SKILL ($180) so buy the Crucial instead. I just happened to get the G.SKILL a little bit cheaper than Crucial when I bought the parts. If you do not need the SSD since you have a NAS (I have the Synology DS1812+) then forget about the links above. In fact, my other ESXi host does not have an HDD or SSD installed in it. Though, that may change in the future. I already have Samsung 850 EVO 1TB SSD in my NAS to speed the VMs up but I might still buy one for Flash Read Cache.

Installation

Before installing ESXi 6.0 Update 2 on the Intel NUC, I suggest checking the BIOS version first. Mine came with BIOS version 24, which was released in Oct 2015. The new version is 36, which was released last month, can be downloaded directly from Intel’s download page. The BIOS will work on all four models so be sure to download and apply it first before doing something else. Upgrading the BIOS was very easy. All I did was download the SY0036.BIO file, copied it to a USB flash drive, and powered on the NUC with the USB flash drive in it and hit the F7 key. Then, the NUC recognized that there was a BIO file then I followed the screen prompts to install the new BIOS.

After the BIOS was complete, I head back to my other computer and tried to create ESXi 6.0U2 ISO using UNetbootin. I’ve used that in the past and have successfully installed the different OS using the bootable USB created from UNetbootin. However, this time around, UNetbootin did not recognize my USB flash drive for some reason. My Windows recognized that there was a USB flash drive but UNetbootin did not. I then decided to use Rufus and created the ESXi bootable USB flash drive and it recognized the USB flash drive just fine.

Next, I booted the NUC with the USB flash drive and entered BIOS settings to make sure that the boot order was correct so that every time that the NUC reboots, it always tries to boot from the USB. Once ESXi installer files were loaded to the RAM, I followed the screen prompts and picked the same USB flash drive I used to boot it from as the destination drive for the ESXi files.

ESXi on Intel NUC

Thoughts

As you can tell from my tweet, I’ve only had the Intel NUC for two days but I am quite pleased with it. I am not going to lie but I wish my NUC had more oomph. I could’ve gotten the Core i5 or wait for the Core i7, which in theory, should provide more oomph but stay under $500 was important. The $90 difference in Core i5 is probably not much but I’ve spent quite a bit of tech stuff for that past few months, like my router, managed switches, and 1TB SSD. If I ever need more compute power, I could use the vMotion feature to move it to my Xeon-based ESXi host and run it from there.

I also noticed that the NIC doesn’t seem to work correctly when I set it to 9K MTU. I kept getting an error message and when I did a quick search one guy said that it means that the NIC does not support the jumbo frame. However, one reader said that the NIC itself supports the jumbo frame but the driver might be the culprit. It’s not really a big deal but it would’ve been great if I could’ve used 9K MTU without doing some tweaking. The majority of my wired devices are already set to 9K so this is another odd man out.

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Damn Small Linux VM For Home Lab

09/01/2015 By Andrew Roderos 1 Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Download my Damn Small Linux OVA here (mirror link). Note: OVA was exported from a VMware ESXi 6.0 host. It might need to be converted for a different hypervisor.

I am getting ready to build my F5 BIG-IP Application Delivery Controller, commonly known as load balancer, home lab and one of the things I need for the lab is to have servers in the internal network side. While I have some Linux VMs already in production, they are not that small compared to Damn Small Linux. I’ve heard and/or read about this distro a long time ago, but never really bothered to check it out until now.

What is Damn Small Linux?

Damn Small Linux is a very versatile 50MB mini desktop oriented Linux distribution. DSL was originally developed as an experiment to see how many usable desktop applications can fit inside a 50MB live CD. It was at first just a personal tool/toy. But over time Damn Small Linux grew into a community project with thousands of development hours put into refinements including a fully automated remote and local application installation system and a very versatile backup and restore system which may be used with any writable media including a USB device, floppy disk, or a hard drive.

VM Hardware Specs

For this install, I used the following:

RAM = 64MB – Using 32MB of vRAM was too slow for my taste.

HDD = 128MB – I didn’t test if I can go smaller using Frugal Install. Make sure to use IDE because SCSI won’t work. One could try to play with boot options to make it work but IDE is fine.

CPU = 1 x vCPU and 1 x core

NIC = 1 x vNIC – Network adapter is set to Flexible

How to install Damn Small Linux?

If one has to create his own version, please follow the steps below. Obviously, change the values for one’s needs.

1. Boot the ISO. You will be greeted like the one on the screenshot. Type dsl 2 on the boot prompt.

first boot

2. Once booted, run cfdisk. Prompt will show something like root@tty1[/]#. Type cfdisk there and hit enter key.

3. A new prompt will show. Type y and hit enter.

No partition table or unknown signature on partition table
Do you wish to start with a zero table [y/N] ?y

4. A new screen will appear.

cfdisk

5. Hit tab to go to the [New] option.

cfdisk new

6. A new screen will show up with three options: Primary, Logical, and Cancel. Select [Primary] and hit enter.

cfdisk primary

7. A new screen will show Size (in MB). Type 75.

cfdisk 75

8. A new screen will show three options: Beginning, End, and Cancel. Select [Beginning] and hit enter.

cfdisk beginning

9. Make hda1 bootable. Select [Bootable] option and hit enter.

cfdisk bootable

10. Select the Free Space partition by hitting the down arrow key. Follow step 6, 7, 8, and 9. For step 8, change it to 25.

11. Change the file system type of hda2 to swap. Tab (or right arrow key) to [Type].

cfdisk hda2 type

12. Type 82 and hit enter.

cfdisk hda2 filesystem

13. Select the Free Space partition by hitting the down arrow key. Follow step 6, 7, 8, and 9. Size will be the remaining one.

14. Once completed, select [Write] option by hitting tab or right arrow and hit enter. A new screen will show up and type yes and hit enter.

cfdisk write yes

15. Exit out of the cfdisk utility. Select [Quit] and hit enter.

16. Prepare the partitions and power off.

root@tty1[/]# mke2fs /dev/hda1
root@tty1[/]# mke2fs /dev/hda3
root@tty1[/]# mkswap /dev/hda2
root@tty1[/]# swapon /dev/hda2
root@tty1[/]# poweroff

17. Once completely powered off, turn it back on and let it boot. May need to manually power it off. By powering it off, it will not eject the ISO on the next boot. Boot will fail without the ISO since the OS is not installed yet. Once booted, one will be greeted by an old fashioned GUI.

first gui boot
install dsl

18. Install Damn Small Linux (DSL). Right click somewhere in the desktop and go to Apps > Tools > Frugal Install > Frugal Grub Install.

19. A terminal window appears. Follow the steps shown on the screenshot or the text below.

frugal grub install
Enter the target partition to hold image (EXAMPLE: hda2): hda1
Install from:
[L]iveCD or Frugal Install
[P]endrive
Local [I]so File
Fetching latest iso from [W]eb
From Poormans via [B]ootfloppy
Choose (l/p/i/w/b): l
Enter partition to be used for MyDSL extensions.(EXAMPLE: hda3): hda3
List boot options:
Example: vga=normal toram tz=US/Pacific ssh fuse: <hit enter>
Choose language/keyboard if other than english:
Example: cs da de es fr nl it pl ru sk: <hit enter>
For INSTALL answer y to format, for UPGRADE answer n.
Format the target partition /dev/hda1 (y/..)? y
Last change to exit before destroying all data on /dev/hda1
Continue (y/..)? y
Formatting /dev/hda1
<-- Output ommitted for brevity -->
Grub Installation Completed.
Press Enter key.

20. Power off the system. Open a terminal and issue sudo poweroff command.

21. Before turning on the VM, remove the ISO to verify that the installation was successful and it boots from the hard disk. Let it boot or just hit enter to boot instantly.

22. DSL X Setup. Go to Cancel option and hit enter.

23. Edit the GRUB menu. Open a terminal window.

dsl@box:~$ sudo su
[/home/dsl]# mount /mnt/hda1
[/home/dsl]# vi /mnt/hda1/boot/grub/menu.lst

Comment out all the title, kernel, and initrd lines. One could technically remove everything but for backup purposes, just comment them out. Once commented out, add a customized DSL. Once added, save and quit.

default 0
timeout 3
title DSL
kernel /boot/linux24 root=/dev/hda1 quiet vga=normal noacpi noscsi frugal dma toram restore=hda3 home=hda3 opt=hda3 host=DSL dsl mydsl=hda3
initrd /boot/minirt24.gz

Sample GRUB menu.lst file.

grub menu

24. Reboot DSL.

25. DSL X Setup is going to pop up again. Just hit cancel.

Optional

The OVA file has all the ones mentioned below so download it to save some time.

Want to stop Dillo browser from starting up every time DSL boots up?

Edit the .xinitrc file. Find the line where it says dillo and comment that line out. Please use the screenshot below as a reference.

dsl@box:~$ sudo su
[/home/dsl]# vi .xinitrc
xinitrc

Want Monkey Web Server to start  automatically during boot up?

Edit bootlocal.sh file. This file is like autoexec.bat in DOS. Add the line below.

[/home/dsl]# vi /opt/bootlocal.sh
/opt/monkey/bin/banana start

Want SSH Daemon to start automatically during boot up?

Edit bootlocal.sh file. Add the line below.

[/home/dsl]# vi /opt/bootlocal.sh
/etc/init.d/ssh start

Thoughts

I spent several hours researching and playing with DSL to get a perfect base VM, but I eventually decided to leave it alone. It’s only going to be used for my F5 load balancer lab, so why spend more time into it. Did I need to do all this? Probably not. I could have cloned a base Ubuntu Server VM with LAMP installed but I really wanted a small Linux VM. It’s perfect for resource constraint machines, not that my ESXi host is, like notebooks.

This is far from perfect DSL install since I couldn’t figure out how to make my password changes permanent. Also, the SSH keys gets generated every boot up so when I connect via SSH, I kept getting new keys every time the server gets rebooted. Yes, one of the links listed below have a way to auto start SSH daemon, but it won’t start because the SSH keys need to be generated. Launching sshd at boot does not necessarily mean it will be successful. No SSH keys, no workie.

References

Frugal Install
Bootlocal.sh
Monkey Web Server

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

CCIE R&S v5 Home Lab

08/16/2014 By Andrew Roderos Leave a Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Cisco announced CCIE Routing & Switching Version 5 exam update back on December 3, 2013. Even before the announcement, there were already speculations on what will or will not be on the exam. There were two things we knew for sure that will not be in the v5 exam, Frame Relay and IOS 12 – we all turned out to be right. That being said, my old CCIE R&S v4 home lab needed an upgrade.

UPDATED: If you are running ESXi 6.x, please look at the updated section below.

Before the CCIE R&S v5 exam has been released, I’ve been studying for the written exam by using the two very well known books for Cisco routing and switching – it does not matter if one is studying for CCNA R&S, CCNP R&S, or CCIE R&S. These two books are very awesome resources to have on their bookshelf at work or at home. The two books are Routing TCP/IP, Vol 1 and Routing TCP/IP, Vol 2. While these two books are old, they are still very helpful! According to Jeff Doyle, he is working on updating at least one of the books. You can now order Routing TCP/IP, Volume II Second Edition!

The other two books that a CCIE aspirant must have are the following: CCIE R&S v5.0 Official Cert Guide, Vol 1 and CCIE R&S v5.0 Official Cert Guide, Vol 2. The first two books are also available in a bundled format, CCIE R&S v5.0 Official Cert Guide Library, which costs less than buying them separately. I do want to point out that these books are not a replacement for Routing TCP/IP Vol 1 and 2.

Cisco’s Recommended Gears

The recommended gears to prepare for the exam are ISR G2 2900s and Catalyst 3560X, as stated in this document. I can’t afford all of those gears, especially with INE’s hardware topology. As you can see, the topology has 20 routers for full-scale labs but the advanced technology labs require only 10 routers. Even with 10 routers, I still can’t afford to buy them. On top of all the routers, you still need the 3560X which costs an arm and leg just like the 2900s. Fortunately, there are other options in building a CCIE home lab that will mimic INE’s workbook topology. I opted for the CSR 1000v which requires a hypervisor, like VMware’s ESXi, Microsoft’s Hyper-V, Xen, and KVM.

The CSR 1000v takes care of the routing section of the lab, but I still need something for the switching section. Fortunately, I can reuse my two Catalyst 3560s loaded with IOS 15. Yes, you read that right. There are some versions of Catalyst 3560v1 that are capable of loading IOS 15 as long as you have the 32MB flash version. I have two flavors of 3560: WS-3560-24TS-S and WS-3560-48TS-S. If I try to log into Cisco’s IOS download page with your CCO account and start looking for an IOS 15 image for that model, you won’t be able to find one. The last version for these models is 12.x, but if you look under 3560G then you have the option to download IOS 15 and the images do work on these two models.

With two on hand, I needed to add two more to complete my CCIE home lab. Fortunately, we have tons of them at work that aren’t being used at all. I wanted to buy them from my employer but was told that I didn’t have to. I was given a permission to borrow two of them for a long period so I took that opportunity and borrowed two more of the WS-3560-24TS-S. With four 3560s, I now have a complete CCIE R&S v5 home lab! While it won’t do 100% of the topics covered in the lab exam, at least I can still do the majority of the topics with my home lab. For the topics that won’t be able to do with my home lab, I can always do rack rental or probably use VIRL (Virtual Internet Routing Lab).

ESXi Server

Since I already have a VMware ESXi home lab, it was an easy decision on which hypervisor to use and it’s what INE is using anyway for their rack rental. The ESXi server build that I have is a couple of years old so if you’re looking for a new one, then check out my new build to get some idea.

Related: VMware ESXi Home Lab – Intel NUC 10 (Frost Canyon)

My new build is underpowered compared to my first ESXi host, but the Intel NUC Skull Canyon is a great candidate for your home lab. If the build is still pricey for you, eBay is there for old servers. I almost bought the Dell T5500 but glad I decided to hold off on the purchase. Also, I didn’t like the fact that it was too big since I do not have space for it unless I clear out my rack. Anyone wants to buy my old routers and switches that were part of my CCIE R&S v4 home lab? The decision to hold off turned out to be a great decision since I was able to squeeze all 20 CSR 1000v with 2.5GB RAM configured on each VM, more on that later.

Installation

I am not going to do a tutorial on how to install it since INE has one already. However, I do want to show some tips that I’ve learned from the INE’s forum members on both CCIE R&Sv5 Equipment Build and Building INE’s RSv5 topology on CSR1000v threads.

INE’s blog post about how to install CSR 1000v is a bit old, but you can still follow the wizard since it’s pretty self-explanatory. One of the differences would be after the Name and Location section, which is new with the OVA file that I downloaded. The Deployment Configuration section will ask you what type of deployment you would like to use – Small, Medium, Large, and Large + DRAM Upgrade. For the lab environment, a small hardware profile is sufficient.

If you still decided to use ESXi 5.5, then deploying the OVF template install is different from ESXi 5.1. Different in terms of how it creates the VM. The ESXi 5.1 used hardware version 9 versus hardware version 10 of ESXi 5.5. You’re probably asking, so what if ESXi 5.5 uses hardware version 10. Well, you cannot edit the VM using the vSphere client. You would need to use vSphere Web Client to edit the VM, as seen below. That means you would need to run a vCenter Server instance to just run vSphere Web Client which complicates your home lab. Then again, if you are studying for VCP or will study for it, then it doesn’t really matter. However, for people who just want to get their CCIE home lab running, then it will be annoying to go through the process of building a vCenter Server instance just to edit the VM.

editing the vm
Editing the VM

Tips

If you still decided to use ESXi 5.5, then there are ways around the error that you get when trying to edit a virtual machine that is version 10 or higher. Some people decided to copy the VMX file from the ESXi server to their desktop and edit the file and re-upload it. I decided to use CLI of the ESXi server since I am familiar with it. To access the CLI, you’ll need to enable SSH, it’s mentioned in one of my blog posts. Next, you will need to find the VMX file of your CSR 1000v. In my case, my baseline VM is called CSR1000v and normally what you call your VM will be also the name of the folder within your datastore.

~ # vi /vmfs/volumes/nfs01/CSR1000V/CSR1000V.vmx
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "10"
nvram = "CSR1000V.nvram"
! Output omitted for brevity !

The line that we’re looking for here is the virtualHw.version = “10”. We need to change it from 10 to 9 and save the file. We’re still not quite done yet, even though we’ve changed the hardware version. The vSphere Client will still give you an error if you try to edit it since it still thinks that the VM is version 10. We need to remove the VM from the inventory and add it back in. Once added, the VM can now be edited. My suggestion is not to edit this and make this as your baseline VM for CSR 1000v needs. Since we need 20 CSR 1000vs for INE’s full-scale labs, we need to start cloning the VM. Once you’ve created all 20 VMs, you’re now ready to edit them.

In INE’s blog post, it tells you to issue the command platform hardware throughput level 50000 but this one doesn’t prompt you to activate the evaluation license with a newer CSR 1000v image, at least on the one I downloaded – csr1000v-universalk9.03.12.00.S.154-2.S-std.ova.

R1(config)#platform hardware throughput level 50000
R1(config)#
*Aug  6 04:36:20.067: %VXE_THROUGHPUT-3-CONF_FAILED: Configuration failed. Installed license does not support the throughput level. Please install the valid license

To activate the evaluation premium license, you need to issue the command below and accept the EULA. Before activating the license, you may want to do a snapshot or configure your VM’s Virtual Disk to non-persistent. This way, you can still take advantage of the higher throughput level compared to a measly 2.5Mbps once the evaluation license expires. For the lab environment, though, the basic throughput is enough so it’s really optional. Another option is to activate the license and put all the initial configs and then convert the VM’s Virtual Disk to non-persistent. This will give you fresh evaluation license every time you the shutdown and power up the VM.

R1(config)#license boot level premium 
         Feature Name:prem_eval
! Output omitted for brevity !
ACCEPT? (yes/[no]):yes
*Aug  6 04:41:05.560: %LICENSE-6-EULA_ACCEPTED: EULA for feature prem_eval 1.0 has been accepted. UDI=CSR1000V:9L5WSSXMWKP; StoreIndex=0:Built-In License Storage% use 'write' command to make license boot config take effect on next boot
*Aug  6 04:41:09.344: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = csr1000v Next reboot level = premium and License = prem_eval
R1#show license
! Output omitted for brevity !                
Index 30 Feature: prem_eval                      
        Period left: 8  weeks 3  days 
        Period Used: 0  minute  3  seconds 
        License Type: Evaluation
        License State: Active, Not in Use, EULA accepted
        License Count: Non-Counted
        License Priority: Low
! Output omitted for brevity !

Once the license is activated, then you can now change the throughput to 50Mbps.

R1(config)#platform hardware throughput level 50000
R1(config)#
*Aug 21 01:34:36.573: %VXE_THROUGHPUT-6-LEVEL: Throughput level has been set to 50000 kbps

What happens when the evaluation license expires? Well, as stated earlier the throughput goes back down to 2.5Mbps but you still get all the premium license features. You also get an annoying message about the license being expired. If you’re doing debugs and what have you then it gets annoying to see one of the lines is about the license. That said, the non-persistent disk or snapshot becomes handy in this situation.

R1#show platform hardware throughput level
The current throughput level is 2500 kb/s
R1#             
%LICENSE-1-EXPIRED: License for feature prem_eval 1.0 has expired 7 hours and 30 minutes ago. UDI=CSR1000V:905F9PPAKYB

Since my ESXi server only has 32GB of RAM (maxed out), the amount of CSR 1000v VMs that I can do is probably around 12 excluding my other VMs. That means I will be short of 8 routers for the full-scale labs. My original plan was to do memory overcommitment and take advantage of the performance that you get from SSDs. That said, I bought a 128GB Samsung SSD to be used as a host SSD cache. In theory, the swapping to SSD shouldn’t be noticeable for a lab environment. I was ready to install the SSD until I saw a guy posted in the IEOC thread about disabling the Transparent Page Sharing (PDF) Large Page support feature. By default, some ESXi versions support large pages and small pages. By disabling the large page support for OS, VMware ESXi kernel will force the guest OS to use small page support and will increase the chances of TPS being able to keep one copy of the page and share it among the VMs. Since we’re running the same OS twenty times over, there are some contents that will be the same for each and every VM. That said, it would make sense to just keep one and share it among the VMs, which reduces the amount of physical memory being used by the VMs and allows a higher level of memory overcommitment. That’s how I was able to run 23 VMs concurrently and use only about 16GB of physical memory.

To disable the large page behavior and instead force small pages, go to Configuration tab > Advanced Settings (under Software) > Mem > Mem.AllocGuestLargePage > Change the value from 1 to 0.

Forcing small page support
Forcing small page support

Once disabled, it would take some time to scan the VMs and reduce the amount of RAM used. That being said, just try to run 10 or so CSR VMs and wait several minutes for the ESXi server to scan them for duplicate memory pages. Once ESXi is able to scan fully, you should start to see your memory consumption to drop significantly.

UPDATE: For those people who are running certain versions of ESXi 5.x and now 6.0, the TPS is disabled by default. My ESXi hosts are now on 6.0U2 6.5 and noticed that Mem.AllocGuestLargePage setting changed back to the value of 1. Please check your settings and change it back to 0. In addition, there is a new setting in certain versions of 5.x and version 6.0 that needs to be changed. This is to revert back to the traditional behavior of TPS. The setting is the Mem.ShareForceSalting and needs to be changed to 0. Please be aware that there is a VMware KB that talks about security concerns of TPS. In a CCIE home lab environment, I do believe that it is perfectly OK to revert back to the traditional behavior.

Inter-VM Transparent Page Sharing

Previously, once the Mem.AllocGuestLargePage has been changed to 0 the inter-VM TPS will kick in after several minutes. However, I noticed that the behavior of ESXi 6.0U2 is quite different. The inter-VM TPS did not kick in until the VMs were powered down or migrated (vMotion) to a different host. Once done, RAM consumption will start to decrease but may take 15 – 30 minutes before it goes down to around 14 GB. Ignore the other VMs on the left since those are on a different host. The RAM consumption of 20 CSR1000v should be around the same as mine.

TPS Enabled
TPS Enabled

Thoughts

If you haven’t been living under a rock, then you definitely know that there are several options out there. Some people decided to use Web IOU or Unified Networking Lab by @adainese, GNS3 v1.x, IOSv that is extracted from the OnePK, or a combination of hardware and virtual. If you are not Cisco employee, then you shouldn’t be running IOU/IOL since it is against Cisco’s EULA. That said, run it at your own risk.

The GNS3, Web IOU, and Unified Networking Lab options are great since you can take them on the go because the system requirements are not very high so any decent notebook can run it. The problem with IOU/IOL, they weren’t designed to be used for learning so some features may not work properly and can be frustrating at times. The IOSv’s, which is the software that will be used in VIRL, system requirements are not very high as well so if you have a not so powerful desktop/notebook, then this is definitely something you can consider.

Since I happened to have an ESXi server, I made a decision to utilize it since I was only using it for playing with some OS and a few VMs that I use for everyday use like FTP, proxy, Plex, etc. Another reason why I went with the CSR 1000v route is because the INE’s labs were written with their rack rental in mind. That said, the convenience that you get with following INE’s setup is priceless. I am quite happy with my setup and my ESXi server can handle the load that I am currently throwing at it. Then again, I haven’t finished the whole advanced technology labs so take it with a grain of salt.

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

VMware ESXi Home Lab [2012]

09/12/2012 By Andrew Roderos 1 Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

If you’ve been following my tweets, then you know that I haven’t been doing a lot of CCIE studying. For about three weeks last month, I tried to finish my Openfiler + ESXi (vSphere Hypervisor) home project. Obviously, it wasn’t three straight weeks worth of time – a few hours per day/night. I’ve been very excited about this home project and I’ve already put off another mini-project that I have to do – which is to install my Samsung 128GB SSD drive and drive caddy to replace my optical drive for extra HDD space on my mid-2009 15″ MacBook Pro.

Anyway, if you’ve been researching for whitebox builds then you’ve probably have seen people using Mac Mini as their ESXi host. I’ve considered running ESXi on Apple Mac Mini since others have successfully done it using this guide, but I’ve decided against it. The form factor of Mac Mini is hands down a good feature to have, but there are some limitations to it that may not work for the future growth of my home network and my need to play with different technologies outside Cisco.

There is also Intel NUC as a very attractive ESXi host. However, the older generation of NUCs needs to have a custom ISO so that the NIC would work. With the new Intel NUC (6th generation), the ESXi 6.0 Update 2 works out of the box on these units. I actually talked about it here since I’ve added another ESXi host in my home lab.

Related: VMware ESXi Home Lab – Intel NUC (Frost Canyon)

Before we delve into the ESXi build, let me give you a little bit of background of my old setup. My old server was supposed to be my ESXi since the motherboard is listed here as compatible with ESXi. However, upon building it and really giving a thought it turned out to be not the best solution for what I wanted it to do. I could make it work but as what other people would say just because you can doesn’t mean you should. Having said that, I just built it as a regular Windows box with virtualization software running on top of it to run VMs with different OS (Linux, Unix, and Windows) in them. It worked great and no complaints. However, as a technology enthusiast, there was this voice that kept telling me to run VMware ESXi at home. As you can already tell, the voice won and I’ve decided to buy and build a new home server. Besides, this particular build will be used for VCP studying if ever I decide to really get serious in pursuing it. I do want to get it, but it isn’t my day-to-day job so I am still contemplating. Yes, I am well aware that you need to have instructor-led training first before you can register to take VCP. I am actually currently enrolled in a local community college which is a member of VMware Academy. This is the cheapest route for me so I decided to sign up for the class. I could’ve asked my current employer to pay for the boot camp style VMware vSphere: Install, Configure and Manage class but I’ve decided against it since it really is outside my day-to-day job responsibilities. I am just taking this class just to expand my knowledge outside the Cisco realm.

Anyway, for curious minds out there, here’s my old home server specs:

AMD Phenom II X6 1090T Black Edition
ASUS M4A88TD-V EVO/USB3
2 x G.SKILL Ripjaws 8GB
Dell PERC 5/i
3 x Western Digital Caviar Black 1.5TB
Corsair AX850 PSU

Without further ado, here are the parts of the home server:

Lian Li PC-V351B
Supermicro MBD-X9SCL+-F
Intel Xeon E3-1230 V2 Ivy Bridge
Antec EA-380D Green
2 x Kingston 8GB DDR3 SDRAM ECC Unbuffered Update: It’s 32GB RAM now.
SanDisk Cruzer Blade 4GB USB Flash Drive

This whitebox build is in the high-end range, so if you are looking for just ESXi home lab use then this may not serve your purpose. There are other builds out there that are suitable for inexpensive home lab use, like HP N40L, so choose whatever will work for you. This build will also be used for ESXi home lab (nested ESXi 5.x) and for “production” so I wanted a more powerful host than the N40L and more expandable in the RAM side of things. As already mentioned, I run different OS with different purposes in my home server to provide services for myself, friends, and family. Some of these services are the following but not limited to proxy, FTP, SSH, and etc.

If you were paying attention, the build didn’t have any hard drives in it. This build was designed to run quietly and with low power requirements. I chose to use 4GB USB flash drive to install ESXi on to save power, money, and repurpose the OS hard drive on my old server. Since I do not have CD/DVD-ROM driver either, I searched the web on how to create a USB bootable flash drive and it was pretty straightforward, just follow this guide.

At first, I used Openfiler as my NAS and it worked great. However, I am convinced that the cause for crashing/rebooting of my NAS box was because my RAID controller was overheating. Never really spent a whole lot of time troubleshooting it because this project went over the projected schedule that I wanted to spend my free time on. With that said, I just pulled the trigger and bought Synology DS1812+ as my NAS for CIFS/SMB and NFS shares that my ESXi box will use as its datastore. So far, I love this Synology NAS box and will never go back to Openfiler. Don’t get me wrong, Openfiler is really good but I believe Synology or other NAS manufacturers out there is the way to go. Yes, I could’ve stuck with Openfiler by just buying a new RAID controller to save cost initially, but over time I believe I’ll end up paying more money on energy costs by using a regular PC.

Well, I hope this blog post is a little bit helpful for the people trying to build a whitebox ESXi host. Let me know if you have any questions and I’ll try to answer them as soon as I can. Enjoy!

You might also like to read

VMware ESXi Home Lab [2016]

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

CCIE R&S v4 Home Lab

10/03/2011 By Andrew Roderos 5 Comments

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Update: New home lab for CCIE R&S v5.

As you may know, I am studying for the CCIE written and lab exam. To help with my CCIE studies, I have decided to build my own CCIE home lab. Sure, you can rent CCIE racks and there are tons of them out there. However, you don’t get the convenience of being able to turn it on and off at your own time. Another alternative of having a rack is building a GNS3 or Dynamips server to run Cisco IOS. It is a very inexpensive way to study for CCIE and it works great for other people. While Cisco uses IOU (IOS on Unix) on their troubleshooting section, they’re not running any emulators for the configuration section. That said, configuring routers and switches on a real gear will mimic the same results. Other CCIE aspirants will disagree on this logic; however, I’ve met and read forum member’s posts who has the same reasoning as I do. Besides, I started building my home lab back in 2005 when I first entered the Cisco Network Academy Program (CNAP). I added more routers and switches to my home lab when I studied for my CCNP exams, so few more routers and switches here and there wouldn’t be such a big deal. Though, it did cost me a lot of money.

Internetwork Expert (INE) was my first choice in CCIE lab preparation. At the time I decided that I am going to pursue CCIE, it was the most talked about CCIE training vendor and all of them were positive. It probably still is, but I haven’t had a chance to visit and read forums lately. That said, I decided to build my lab as close as possible with what they use for their CCIE workbooks since they have a specific topology for their CCIE workbooks. Fortunately, the hardware specifications and topology for the CCIE workbooks are open for public and can be found here. As mentioned earlier, I started building my home lab back in my CCNA days. That said, the gears are quite different compared to INE’s specs. Having different equipment turned out to be a challenge because the interfaces are different. I find myself trying to figure out which interface is which when I am doing labs. That said, I created my own topology with the right interfaces so I can at least save some time when I am labbing.

CCIE Home Lab

I think I should stop talking now and give you an idea of how my CCIE home lab looks like. Here’s the list of my CCIE home lab equipment:

DevicePlatformDRAMFlashWIC(s)Software
Backbone 1 (BB1)2620XM128MB48MB1 x NM-8A/SAdvanced Enterprise Services 12.4(18)A
Backbone 2 (BB2)252016MB16MBN/A12.2(15)T17
Backbone 3 (BB3)252016MB16MBN/A12.2(15)T17
Router 1 (R1)2620XM128MB48MB1 x WIC-2TAdvanced Enterprise Services 12.4(18)A
Router 2 (R2)2620XM128MB48MB1 x WIC-2TAdvanced Enterprise Services 12.4(18)A
Router 3 (R3)2620XM128MB48MB2 x WIC-2TAdvanced Enterprise Services 12.4(18)A
Router 4 (R4)2801256MB64MB1 x WIC-2TAdvanced Enterprise Services 12.4(24)T4
Router 5 (R5)2801256MB64MB1 x WIC-2TAdvanced Enterprise Services 12.4(24)T4
Router 6 (R6)2801384MB128MB1 x WIC-2TAdvanced Enterprise Services 12.4(24)T4
Switch 1 (SW1)3560-24TS128MB32MBN/AEnhanced Multilayer Image (EMI) 12.2(44)SE
Switch 2 (SW2)3560-48TS128MB32MBN/AEnhanced Multilayer Image (EMI) 12.2(44)SE
Switch 3 (SW3)3550-48TS64MB16MBN/AEnhanced Multilayer Image (EMI) 12.2(25)SEC2
Switch 4 (SW4)3550-48TS64MB16MBN/AEnhanced Multilayer Image (EMI) 12.2(25)SEC2
Terminal Server251116MB16MBN/A12.2(15)T17

The image below is the picture of my home lab without the cables.

The image below is the Frame Relay topology.



The image below is the Ethernet connectivity topology.

I hope I can inspire more network engineers to pursue Cisco’s highly coveted and prestigious CCIE Routing & Switching certification. It will take a lot of your time and money, but it will be all worth it once you get your own five-digit number! Heck, by just studying for the exam will make you a better network engineer.

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Footer

WORK WITH US

Schedule a free consultation now!

LET’S TALK

Copyright © 2011–2023 · NetworkJutsu · All Rights Reserved · Privacy Policy · Terms of Use