Creating a Splunk Lab with AutomatedLab

It’s been a while since I have posted, but it’s also been a while since I’ve worked with AutomatedLab.  I’ve been busy moving to Seattle over the past couple of months.  I’m back now and I’ve got two labs to create – the first is for the Splunk Windows Infrastructure app and the second is for the Splunk Exchange app.  Let’s start with the easier one – Windows Infrastructure.​​​

​The header for this script is similar to all the other AutomatedLab scripts – it’s kind of a recipe at this point:

### Lab Creation Script
$Start = Get-Date

$LabName = "winfra"
$LabDNS = "winfra.local"
$Network = "172.16.50"
$AdminPW = "P@ssw0rd"

# This is where you want the Lab virtual machines to be installed
$VMPath = "D:\Hyper-V"

# This is where the ISO images are located
$ISOPath = "D:\ISO"

# This is installed with AutomatedLab and you shouldn't have to change it
$LabSources = "C:\LabSources"


$LabIndicator = $LabName.ToString().ToUpper();
$LabPath = "$VMPath$LabName"
$ToolsPath = "$LabSourcesTools"

if (-not (Test-Path $LabPath)) {
  Write-Host "Creating Lab Directory $LabPath"
  New-Item $LabPath -ItemType Directory

Write-Host "$LabIndicator: Creating Lab Definition $LabName"
New-LabDefinition -Path $LabPath -VmPath $LabPath -Name $LabName -ReferenceDiskSizeInGB 60

Write-Host "$LabIndicator: Creating Network Definition with $Network.1/24"
Add-LabVirtualNetworkDefinition -Name $LabName -IpAddress "$Network.1" -PrefixLength 24

Write-Host "$LabIndicator: Adding OS Images"
Add-LabIsoImageDefinition -Name Win2012R2 -Path "$ISOPath\en_windows_server_2012_r2_with_update_x64_dvd_4065220.iso" -IsOperatingSystem

Write-Host "$LabIndicator: Administrator Credentials are Administrator / $AdminPW"
$Creds = New-Object PSCredential('Administrator', ($AdminPW | ConvertTo-SecureString -AsPlainText -Force))

Write-Host "$LabIndicator: Adding Domain Definition for domain $LabDNS = NetBIOS $LabName"
Add-LabDomainDefinition -Name $LabDNS -AdminUser 'Administrator' -AdminPassword $AdminPW

​I need five machines for this lab.  Firstly, I need my AD environment – 2 domain controllers running at Windows Server 2012 R2.  I need a manager machine and I need two Splunk machines – one will act as purely an indexer and the other as a search head, license master and deployment server.  Creating servers is relatively easy.  I’m using the root domain prep scripts with slight modifications to create lots of users.

$DC1Activities = @()
$DC1Activities += Get-LabPostInstallationActivity -ScriptFileName PrepareRootDomain.ps1 -DependencyFolder "$PSScriptRootPrepareRootDomain"
$DC1Activities += Get-LabPostInstallationActivity -ScriptFileName New-ADLabAccounts.ps1 -DependencyFolder "$PSScriptRootAddDomainUsers"
$DC1Activities += Get-LabPostInstallationActivity -ScriptFileName InstallSplunkUF.ps1 -DependencyFolder "$PSScriptRootInstallSplunkUF"
$DC1Roles = Get-LabMachineRoleDefinition -Role RootDC @{ DomainFunctionalLevel = 'Win2012R2'; ForestFunctionalLevel = 'Win2012R2' }
Write-Host "$LabIndicator>>> Adding WS2012 Standard (Core) Definition for DC1"
Add-LabMachineDefinition -Name DC1 -Processors 2 -MemoryInMb 512 -DiskSizeInGb 60 -Network $LabName `
    -IPAddress "$Network.2" -DnsServer1 "$Network.2" -DnsServer2 "$Network.3" -DomainName $LabDNS `
    -Roles $DC1Roles -IsDomainJoined -PostInstallationActivity $DC1Activities `
    -InstallationUserCredential $Creds -OperatingSystem 'Windows Server 2012 R2 SERVERSTANDARD' `
    -ToolsPath $ToolsPath -UserLocale en-US -TimeZone 'Pacific Standard Time'

$DC2Activities = @()
$DC2Activities += Get-LabPostInstallationActivity -ScriptFileName InstallSplunkUF.ps1 -DependencyFolder "$PSScriptRootInstallSplunkUF"
$DC2Roles = Get-LabMachineRoleDefinition -Role DC
Write-Host "$LabIndicator>>> Adding WS2012 Standard (Core) Definition for DC2"
Add-LabMachineDefinition -Name DC2 -Processors 2 -MemoryInMb 512 -DiskSizeInGb 60 -Network $LabName `
    -IPAddress "$Network.3" -DnsServer1 "$Network.2" -DnsServer2 "$Network.3" -DomainName $LabDNS `
    -Roles $DC2Roles -IsDomainJoined -InstallationUserCredential $Creds `
    -OperatingSystem 'Windows Server 2012 R2 SERVERSTANDARD' `
    -ToolsPath $ToolsPath -UserLocale en-US -TimeZone 'Pacific Standard Time' `
    -PostInstallationActivity $DC2Activities

$SPLUNKIDXActivities = @()
$SPLUNKIDXActivities += Get-LabPostInstallationActivity -KeepFolder -ScriptFileName InstallSplunk.ps1 -DependencyFolder "$PSScriptRootInstallSplunk"
Write-Host "$LabIndicator>>> Adding WS2012 Standard (Core) Definition for SPLUNKIDX"
Add-LabMachineDefinition -Name SPLUNKIDX -Processors 4 -MemoryInMb 512 -DiskSizeInGb 60 -Network $LabName `
    -IPAddress "$Network.4" -DnsServer1 "$Network.2" -DnsServer2 "$Network.3" -DomainName $LabDNS `
    -IsDomainJoined -InstallationUserCredential $Creds -ToolsPath $ToolsPath -UserLocale en-US `
    -OperatingSystem 'Windows Server 2012 R2 SERVERSTANDARD' -TimeZone 'Pacific Standard Time' `
    -PostInstallationActivity $SPLUNKIDXActivities

$SPLUNKActivities = @()
$SPLUNKActivities += Get-LabPostInstallationActivity -KeepFolder -ScriptFileName InstallSplunk.ps1 -DependencyFolder "$PSScriptRootInstallSplunk"
$SPLUNKActivities += Get-LabPostInstallationActivity -ScriptFileName InstallSplunkDS.ps1 -DependencyFolder "$PSScriptRootInstallSplunkDS"
Write-Host "$LabIndicator>>> Adding WS2012 Standard (Core) Definition for SPLUNK"
Add-LabMachineDefinition -Name SPLUNK -Processors 4 -MemoryInMb 512 -DiskSizeInGb 60 -Network $LabName `
    -IPAddress "$Network.5" -DnsServer1 "$Network.2" -DnsServer2 "$Network.3" -DomainName $LabDNS `
    -IsDomainJoined -InstallationUserCredential $Creds -ToolsPath $ToolsPath -UserLocale en-US `
    -OperatingSystem 'Windows Server 2012 R2 SERVERSTANDARD' -TimeZone 'Pacific Standard Time' `
    -PostInstallationActivity $SPLUNKActivities

$MANAGERActivities = @()
$MANAGERActivities += Get-LabPostInstallationActivity -ScriptFileName InstallSplunkUF.ps1 -DependencyFolder "$PSScriptRootInstallSplunkUF"
Write-Host "$LabIndicator>>> Adding Windows 2012R2 Standard Definition for MANAGER"
Add-LabMachineDefinition -Name MANAGER -Processors 2 -MemoryInMb 512 -DiskSizeInGb 60 -Network $LabName `
    -IPAddress "$Network.6" -DnsServer1 "$Network.2" -DnsServer2 "$Network.3" -DomainName $LabDNS `
    -IsDomainJoined -InstallationUserCredential $Creds -ToolsPath $ToolsPath -UserLocale en-US `
    -OperatingSystem 'Windows Server 2012 R2 SERVERSTANDARD' -TimeZone 'Pacific Standard Time' `
    -PostInstallationActivity $MANAGERActivities

You will note that I have specific activities for InstallSplunk and InstallSplunkUF.  Let’s take a side-bar right now and discuss those.  The Universal Forwarder (InstallSplunkUF.ps1) is the easier one.  I just place the splunkforwarder.msi for the version I want to install in the InstallSplunkUF directory together with an appropriate script:

### Install Splunk from the accompanying splunk.msi
$MSIFile = "$PSScriptRoot\splunkforwarder.msi"

Write-Host "Starting quiet install of $MSIFile"
$st = Get-Date
Start-Process -FilePath $MSIFile -Wait -Verbose `
-ArgumentList @( "AGREETOLICENSE=Yes /Liwem! C:splunk-install.log" )

### Don't forget to set the deploy-poll and add forward-server in your main lab setup

$en = Get-Date
Write-Host "Installation of $MSIFile took $($en-$st)"

The InstallSplunk post-install activity is similar.  However, I have an additional aim – I want to pre-set the admin password and then ensure that the user is not prompted to change the admin password on login.  This requires an additional file – user-seed.conf – that looks like this:

USERNAME = admin
PASSWORD = P@ssw0rd

Also, the script is slightly different.

### Install Splunk from the accompanying splunk.msi
$MSIFile = "$PSScriptRoot\splunk.msi"

Write-Host "Starting quiet install of splunk.msi"
$st = Get-Date

Start-Process -FilePath $MSIFile -Wait -Verbose -ArgumentList "AGREETOLICENSE=Yes LAUNCHSPLUNK=0 /Liwem! C:splunk-install.log"
Copy-Item -Path "$PSScriptRoot\user-seed.conf" -Destination "C:\Program Files\Splunk\etc\system\local"

# Hack to get rid of the UI Login Change Password prompt
New-Item -ItemType File -Path "C:\Program Files\Splunk\etc\.ui_login"

$en = Get-Date
Write-Host "Installation of splunk.msi took $($en-$st)"

I don’t do a lot within these scripts because there are many configuration options and I like to re-use the scripts.  I’d rather do the configuration in my main script – more on that later.  I also have another script called InstallSplunkDS – this adds a bunch of files to the splunk instance I am using for the deployment server and license master – it’s a basic copy operation.  Now, back to our main script​.  Our next step is to do the normal AutomatedLab machine creation.  Since my Splunk Indexer is going to store a bunch of data, let’s create a new virtual disk for it – we’ll configure it later.  I also want to set up our MANAGER machine with the remote system tools.

Write-Host "$LabIndicator>>> Exporting Lab Definition"
Export-LabDefinition -Force -ExportDefaultUnattendedXml
Write-Host "$LabIndicator>>> Setting up Remoting"
Write-Host "$LabIndicator>>> Importing Lab Definition"
Import-Lab -Path (Get-LabDefinition).LabFilePath

Write-Host "$LabIndicator>>> Basic Lab Configuration"
Install-Lab -NetworkSwitches -BaseImages -VMs
Write-Host "$LabIndicator>>> Creating Lab - Domains"
Install-Lab -Domains

if (Test-Path "$LabPath\SPLUNKIDX-Data.vhdx") {
    Remove-Item -Force "$LabPath\SPLUNKIDX-Data.vhdx"
Write-Host "$LabIndicator>>> Preparing Splunk Data Disk on SPLUNKIDX"
Write-Host "$LabIndicator>>> Creating new 100Gb VHD for SLUNKIDX"
New-VHD -Path "$LabPath\SPLUNKIDX-Data.vhdx" -SizeBytes 100GB -Dynamic
Write-Host "$LabIndicator>>> Connecting New VHD to SPLUNKIDX VM"
Add-VMHardDiskDrive -VMName SPLUNKIDX -Path "$LabPath\SPLUNKIDX-Data.vhdx"

Write-Host "$LabIndicator>>> Creating Lab - Other Machines"
Install-Lab -StartRemainingMachines
Write-Host "$LabIndicator>>> Sleeping for 3 minutes, just because it is required"
Start-Sleep -Seconds 180

Write-Host "$LabIndicator>>> Adding Features to MANAGER"
Install-LabWindowsFeature -Name MANAGER -FeatureName RSAT, GPMC, RSAT-DNS-Server

Write-Host "$LabIndicator>>> Doing Post Installs"
Install-Lab -PostInstallations

Finally, we are onto Splunk configuration.  This is where the new features of AutomatedLab v2.5 comes in – we can do powershell invocation directly within our script.  This gives us two ways to configure things – one is with a script in a directory and that’s useful when you need to transfer other things (like the MSI files) to the remote server.  The second way is to run powershell directly, and that gives us the ability to configure things separately.   My first thing to do is to format and mount the additional drive on SPLUNKIDX.  But there are additional things I want to do – I need to adjust where the databases are stored, start Splunk (auto-accepting the license agreement), set up a listener, pre-create indices and disable the Windows firewall.

Write-Host "$LabIndicator>>> Format and Mount Data Drive for SPLUNKIDX"
Invoke-LabPostInstallActivity -ComputerName SPLUNKIDX -UseCredSsp -ScriptBlock {
    Get-Disk |
       Where PartitionStyle -eq 'raw' |
       Initialize-Disk -PartitionStyle MBR -PassThru |
       New-Partition -DriveLetter I -UseMaximumSize |
       Format-Volume -FileSystem NTFS -NewFileSystemLabel "Disk2" -Confirm:$false

    # Adjust the DB location to be I:Splunk
    Write-Host "SPLUNKIDX>>> Creating Database Path"
    New-Item -Type Directory -Path "I:SPLUNK"
    ("`nSPLUNK_DB=I:SPLUNK") | Out-File -Encoding ascii -Append -FilePath "C:\Program Files\Splunk\etc\splunk-launch.conf"

    # Start Splunk with accept-eula
    Write-Host "SPLUNKIDX>>> Starting Splunk Instance"
    Set-Alias splunk "C:\Program Files\Splunk\bin\splunk.exe"
    splunk start --accept-license --answer-yes --no-prompt

    # Enable receiving on port 9997
    Write-Host "SPLUNKIDX>>> Enabling Listener on port 9997"
    splunk enable listen 9997 -auth admin:P@ssw0rd

    # Add indices
    @( "wineventlog", "windows", "perfmon" ) | Foreach-Object {
        Write-Host "SPLUNKIDX>>> Adding Index $_"
        splunk add index $_ -homePath "I:\SPLUNK\$_" -coldPath "I:\SPLUNK\cold\$_" -thawedPath "I:\SPLUNK\thawed\$_" -auth admin:P@ssw0rd

    # Disable the firewall
    Write-Host "SPLUNKIDX>>> Disabling the firewall"
    Set-NetFirewallProfile -Name domain -Enabled False

The set up for the SPLUNK machine is simpler.  We have placed the deployment server and license files in the InstallSplunkDS area, so we don’t need to do anything there.  We do need to start Splunk and add a forward-server, which will require us to restart Splunk.  We will also need to turn off the Windows firewall.

Write-Host "$LabIndicator>>> Configuring SPLUNK"
Invoke-LabPostInstallActivity -ComputerName SPLUNK -UseCredSsp -ScriptBlock {
    # Start Splunk with accept-eula
    Write-Host "SPLUNK>>> Starting Splunk Instance"
    Set-Alias splunk "C:\Program Files\Splunk\bin\splunk.exe"
    splunk start --accept-license --answer-yes --no-prompt

    # Configure Forwarding on this box
    Write-Host "SPLUNK>>> Forwarding all events to SPLUNKIDX:9997"
    splunk add forward-server SPLUNKIDX:9997 -auth admin:P@ssw0rd

    # Restart splunk
    Write-Host "SPLUNK>>> Restarting"
    splunk restart

    # Disable the firewall
    Write-Host "SPLUNK>>> Disabling the Password"
    Set-NetFirewallProfile -Name domain -Enabled False

A quirk of Splunk is that certain things need to be done in a certain order – in this case, we need the Splunk indexer up to add the forward server, and we need the Splunk license master up to add the license server, and we need the indexer to be licensed to set up distributed search so we have to do two service restarts.  Here is the rest of the configuration.

Write-Host "$LabIndicator>>> Configuring SPLUNKIDX Licensing"
Invoke-LabPostInstallActivity -ComputerName SPLUNKIDX -UseCredSsp -ScriptBlock {
    # Set up license master on splunk (needs to be done after SPLUNK is set up)
    Write-Host "SPLUNKIDX>>> Setting up License Slave"
    Set-Alias splunk "C:\Program Files\Splunk\bin\splunk.exe"
    splunk edit licenser-localslave -master_uri https://SPLUNK:8089 -auth admin:P@ssw0rd

    # Restart splunk to install license
    Write-Host "SPLUNKIDX>>> Restarting"
    splunk restart

Write-Host "$LabIndicator>>> Configuring SPLUNK Distributed Search"
Invoke-LabPostInstallActivity -ComputerName SpLUNK -UseCredSsp -ScriptBlock {
    # Set up the distributed search peer
    Write-Host "SPLUNK>>> Setting up distributed search"
    Set-Alias splunk "C:\Program Files\Splunk\bin\splunk.exe"
    splunk add search-server -host SPLUNKIDX:8089 -auth admin:P@ssw0rd -remoteUsername admin -remotePassword P@ssw0rd

Our final step is to configure the Universal Forwarders.  I could have set up a deployment server in the InstallSplunkUF.ps1, but I wanted to keep those scripts very basic so I can do the remote configuration.  Well, now I’ve completed the splunk instance configuration I can continue on to the universal forwarder configuration.  All the other machines are universal forwarders so I can handle them in bulk:

# Configure the deloyment client and forward server on all Universal Forwarders
# The Where-Object explicitly removes SPLUNK and SPLUNKIDX from this!
Get-LabMachine -All | Where-Object { $_.Name -notlike "SPL*" } | Foreach-Object {
    Write-Host "$($_.Name)>>> Configuring Forwarding and Deployment"
    Invoke-LabPostInstallActivity -ComputerName $_.Name -UseCredSsp -ScriptBlock {
        Set-Alias splunk "C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe"
        Write-Host "Configuring Forwarder"
        splunk add forward-server SPLUNKIDX:9997 -auth admin:changeme</code>

        Write-Host "Configuring Deployment Server"
        splunk set deploy-poll SPLUNK:8089 -auth admin:changeme

        Write-Host "Restarting Splunk UF"
        Restart-Service SplunkForwarder

Wrapping up our script, let’s do a snapshot of the entire system:

Write-Host "$LabIndicator>>> Creating Snapshots"
Checkpoint-LabVM -All -SnapshotName "SetupComplete"

$end = Get-Date
Write-Host "$LabIndicator>>> Setting up the lab took $($end - $start)"

Of course, there is setup still to do for a fully functional Windows Infrastructure Lab.  However, this has the basics of everything installed for you.