TIL: Override puppet from always pulling facts from puppetdb

I never quite understood how puppet always kept pulling facts from `puppetdb` when I ran `puppet facts show` even though I wanted to run the facts locally, like `facter -p`, but `facter` couldn't see my non default module path where all my local facts were

I tried the options `--terminus facter` per the documentation but still `puppetdb` kept being used. In the end, I had a hack that would force local facts to run, but I was never happy with it:

```

puppet facts find $(uuidgen) --modulepath /path/to/modules

```

I ended up digging away for a few hours, even asking the AI and got taken in circles. I ended up asking a competing AI the same question and it pointed me in the right place immediately. It seems if you have `storeconfigs` set true for `puppetdb`, it seems the puppet "redirector code" overrides all other options and consideration and assumes you really want to use `puppetdb`

So, the solution? To turn off store configs when you wanted facts to be calculated immediately

``` puppet facts show --no-storeconfigs --modulepath /path/to/modules

```

Also got the added bonus of `puppet facts show` (vs `find`) is that it has the `--timing` option to test how long each fact takes to run

Squid configuration in 2024

Configuring Squid in 2024 should be easy, but with over 20 years worth of posts describing problems and solutions and the ongoing evolution of squid means it can be difficult to work out what configuration option you need and how to configure it. Especially for SSL

SSL support for squid made some serious strides in versions 3 and 4 and has settled down after that, yet it can be unclear the correct configuration to use with many posts from that transition period of versions 3 to 4

For example, there are so many posts about configuring SSLBump or Peek and Slice, that you can become easily confused. All that sort of external configuration is now taken care of internally by modern versions of squid, and the most configuration you may need to do is to generate host certificates

In this example, the (internal) parent listens on port 443 and will redirect any port 80 traffic to 443. The squid server is inside a Kubernetes statefulset that redirects port 80 traffic to the pod on port 3128, and 443 to 3129 since the pod does not run as root, so cannot listen on privileged ports

acl localnet src 10.0.0.0/8
cache_peer parent.example.com parent 443 0 no-query default ssl name=myAccel no-digest tls-cert=/etc/squid/certs/tls.crt tls-key=/etc/squid/certs/tls.key
cache_peer_access myAccel allow localnet
cache_peer_access myAccel deny all
http_port 3128 accel defaultsite=parent.example.com no-vhost
https_port 3129 accel defaultsite=parent.example.com no-vhost generate-host-certificates=on tls-cert=/etc/squid/certs/tls.crt tls-key=/etc/squid/certs/tls.key
sslcrtd_program /usr/lib64/squid/security_file_certgen -s /var/cache/squid/ssl_db -M 20MB
sslproxy_cert_error allow all

The only other interesting configuration is the squid server is part of a Load Balanced Domain Name / DNS Traffic Control service and so a certificate is created with Kubernetes certificate manager and the commonName set to the FQDN of the LBDN.

TIL - load testing or benchmarking client limits

 Was load testing a puppet forge implementation today and was hitting some odd errors when I ran each load test immediately after the previous one finished. The errors would not happen if I waited a few (maybe 5 minutes) between tests.

This was odd behaviour but digging through google with the client error of Failed to open TCP connection to (Cannot assign request, I landed on this stack overflow entry (https://stackoverflow.com/a/31877033/14784297) which implied the error mostly came from a lack of ephemeral ports (correct) and more interestingly that after a TCP connection is closed, the connection is still up in a TIME_WAIT state for about 2 minutes:

The reason of this problem is that for opening a TCP connection, the operating system allocates an ephemeral port (for the source port). It binds the socket to the allocated port. After the TCP connection is closed, the connection is left in TIME_WAIT state, typically for 2 minutes, due to historical reasons

Count the number of open connections:

netstat -naptu | grep -c TIME_WAIT

 Reminder - VS Code - simultaneous edits  (aka Multiple selections / multi cursor)

TL;DR shift command L 

I always see VS code highlighting the same key words while editing, but forget how to use that highlighting for simultaneous edits. So I resort to find and replace

The documentation for multiple selections states

VS Code supports multiple cursors for fast simultaneous edits

 But to me, it's really unclear how you then start editing the selected words. On a mac - it is shift command L (⇧⌘L)

 Unifi 7.2.94 on MacOS Error When Saving Settings

After finally getting around to patching my mac mini, everything came up except for the Unifi network application. So, I downloaded the latest version and it failed to start up, so I uninstalled and reinstalled the application (Unifi 7.2.94 on MacOS Catalina 10.15.7) from https://www.ui.com/download-software/

After restoring from a backup, I tried to make the backups run weekly, and got an error message when trying to save settings

An error occurred when applying changes to System settings. This action could not be completed. Please ensure you've entered all information correctly.

Removed, reinstalled application and Java

Java install instructions from https://community.ui.com/questions/Shell-script-for-adding-Java-RE-path-to-local-UniFi-controller-on-macOS/adb1194c-cca4-4a50-8e79-cf805fdd70b3 but Java temurin8 is no longer available so used openjdk@8

$ brew install openjdk@8
$ sudo ln -s /Library/Java/JavaVirtualMachines/openjdk-8.jdk/ /Applications/UniFi.app/Contents/PlugIns/ 
$ sudo /usr/libexec/PlistBuddy -c "Add :JVMRuntime string 'openjdk-8.jdk'" /Applications/UniFi.app/Contents/Info.plist

That didn't help, but the genius in this Ubiquiti Unifi forum link - https://community.ui.com/questions/Cannot-change-save-system-settings-Unifi-Network-on-cloudkey-gen-2-pro/916daacb-39cc-4ecf-a09a-c07fee7cc23d was a cloud key error, but the solution was the same:

Switch back to Legacy Interface (Settings > System > Legacy System > Enable), make any change, then switch back to new Interface (Settings > User Interface > New User Interface > Apply Changes)

Thanks to Corey Quinn for motivating me to publish this just in case anyone else out there has this same problem - https://www.lastweekinaws.com/blog/the-harrowing-search-for-the-elusive-technical-answer/

 Reminder - Low Latency Linux Kernel Boot Options

Terrible name for the web site https://make-linux-fast-again.com/

noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off

Covers many kernel versions as noted in Hacker News article - https://news.ycombinator.com/item?id=25668990

Today I learned - Dell Server Warranty Lookups

In the past, Dell made it hard to access their API for warranty lookups. Even the signup process at https://developer.dell.com/ to get a key was dire - once, they said having 1000 servers wasn't enough to justify API access!

This gist has a working example if you have an API key - https://gist.github.com/teroka/0720274b87b77fe7171f (as well as takes of woe) dealing with the webiste

I also found out that this set of URLs is the one company and it sort of should have a paid option, but doesn't, so I don't understand how it works. I assume they have a Dell API key, or they screen scrpae:

• http://www.updatewarranty.com
• http://www.lookupwarranty.com

$ /usr/bin/curl -s --max-time 30 -H "Referer: http://www.lookupwarranty.com/" 'http://www.lookupwarranty.com/updatewarranty/server/lookup?serviceTag=STAGXXX&modelNumber=&mfg=&email=&platform=Website&key' | jq .

{

  "serviceTag": "HT9K6R3",

  "mfg": "Dell",

  "expires": 1234567890,

  "shipped": 1234567890,

  "outOfWarranty": null,

  "error": null,

  "url": "http://www.dell.com/support/my-support/us/en/19/product-support/servicetag/STAGXXX"

}

Today I Learned - Power On Linux Server at a Particular Time

So today I learned that it is possible to set an alarm in the future for the motherboard to wake up and a poweroff - and the ability has been around for years (since around 2000)!

This article - https://www.linux.com/training-tutorials/wake-linux-rtc-alarm-clock/ - gives a great demonstration on how to set real time clock wakeup by playing with /sys/class/rtc/rtc0/wakealarm. Some good pointers on cleaning out any BIOS based wake ups as well

This article - https://www.maketecheasier.com/alarm-automatically-power-on-linux/ - introduces the command line tool rtcwake which is a little more user friendly

BIND script to create PTR records

BIND script to create PTR records

Just in case you ever need to manage an instance of BIND manually, you will have the pain of remembering to update the PTR records. Here's a quick script to create the PTR records

For those of you that are looking for an alternative, I suggest you use PowerDNS as a backend and PowerDNSAdmin as a nice front end

Here's the script

#!/bin/sh

NETWORKS=$(cat << ENDNET
10.10.1
10.10.2
.
.
.
10.10.99
ENDNET
)

REV_HEADER=$(cat <;
; BIND reverse data file for NETWORK
; Do not edit manually, but run /etc/bin/zones/$0
; after editing /etc/bin/zones/mydomain.tld.db
;
\$TTL    604800
@       IN      SOA     mydns-server.mydomain.tld. root.mydns-server.mydomain.tld. (
                         SERIAL         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      mydns-server.mydomain.tld.
END
)

for NETWORK in $NETWORKS ; do
    echo $NETWORK
    NEWSERIAL=`grep Serial db.${NETWORK} | awk '{printf "%6s", $1+1}'`
    echo $NEWSERIAL
    echo "$REV_HEADER" | sed -e "s/SERIAL/$NEWSERIAL/" -e "s/NETWORK/$NETWORK/" > db.${NETWORK}.new
    egrep "$NETWORK\." mydomain.tld.db | grep -v '^;' | awk 'split($4, ipaddr, ".") {printf "%-3s     IN      PTR     %s.mydomain.tld.\n", ipaddr[4], $1}' | sort -n >> db.${NETWORK}.new
    mv db.${NETWORK}.new db.${NETWORK}
done

New devices and updating their names with lspci

New devices and updating their names with lspci

We just received a new Exablaze X25 card to review and found lspci was reporting the device as "Exablaze Device 0009"

I just found out that lspci doesn't interrogate the card for the name, but uses a lookup table that can be updated with the update-pciids command, which simply downloads from http://pciids.sourceforge.net/v2.2/pci.ids (on my Ubuntu 16 box), but https://github.com/pciutils/pciids says it takes it from http://pci-ids.ucw.cz/pci.ids.

And here is my X25 card as device 0009:

1ce4  Exablaze
 0001  ExaNIC X4
 0002  ExaNIC X2
 0003  ExaNIC X10
 0004  ExaNIC X10-GM
 0005  ExaNIC X40
 0006  ExaNIC X10-HPT
 0007  ExaNIC X40
 0008  ExaNIC V5P
 0009  ExaNIC X25

In InSpec You Do not Need a resource to Test

Am using Chef's inspec to validate a server's configuration, and all the examples show you using a resource and testing against a specific set of tests for that resource.

But I am pulling in a YAML file, and want to validate some of the contents (not the YAML itself), so will do that with ruby and return a number of ruby variables I want to test.

The describe block (or here, describe/subject) can just refer to a variable and not an InSpec resource. Here we are checking dups should be an empty array:

control 'validate-nic' do
  impact 0.7
  title 'Validate nic variable structure'
  paths = nic.map{ |k,v| v.dig('path') }
  dups = paths.select{ |e| paths.count(e) > 1 }.uniq

  describe "Check for duplicate path names in nic" do
    subject { dups }
    it { should be_empty }
  end
end

And that works:

Success:
  ✔  validate nic variable structure: Validate nic variable structure
     ✔  Check for duplicate path names in nic should be empty

Failure:

  ×  validate-nic: Validate nic variable structure
     ×  Check for duplicate path names in nic should be empty
     expected `["nic-2-path-a"].empty?` to return true, got false

Puppet 6 - pulling a value from a hiera hash

My favourite data structure is a hash of hashes (of hashes ...)

In Puppet 6, the Hiera replacement lookup allows you access values by defining the key with dot notation. In previous versions you need to extract the whole hash into a manifests and then extract the data from there

Even better you can do that lookup within Hiera itself

profile::private_ssh_keys:   
  '/home/user/.ssh/id_rsa':
    mode: '0600'
    owner: 'user'
    group: 'groupname'
    content: "%{lookup('keybase::sshkeys.my@keyname.private_key')}\n"

Seem more in the Interpolation Functions documentation - https://puppet.com/docs/puppet/6/hiera_merging.html#interpolation_functions

Jenkins Matrix Based Security with Groovy Scripts

So I think I have just wasted a good day or so trying to work out Jenkins 2.x and matrix based authentication via a groovy script

Some googling gives me some good idea on running groovy scripts, but most are just managing the "Overall" section - like Securing jenkins using groovy and chef.

This gave me a framework to get started, but I couldn't work out a pattern for the Credentials/Agent/Job/Run/View/SCM sections. Then this blog post - Automating authentication and authorization on Jenkins - helped me understand the different level names, but it didn't quite fit

I can't quite see how to map permission names to what was available - how to uppercase, what path does the permission live down??

Here's the code - achieved by trial and error - not much understanding...


// http://blog.albertoviana.com/tag/groovy/


import jenkins.model.*
import hudson.security.*
import com.cloudbees.plugins.credentials.*


def instance = Jenkins.getInstance()
def strategy = new GlobalMatrixAuthorizationStrategy()


// Roles based on https://wiki.jenkins-ci.org/display/JENKINS/Matrix-based+security
//Overall - http://javadoc.jenkins-ci.org/jenkins/model/Jenkins.html
strategy.add(Jenkins.ADMINISTER, 'system-account')
strategy.add(Jenkins.RUN_SCRIPTS, 'system-account')
strategy.add(Jenkins.READ, 'system-account')


strategy.add(Jenkins.ADMINISTER, 'admingroup')
strategy.add(Jenkins.RUN_SCRIPTS, 'admingroup')
strategy.add(Jenkins.READ, 'admingroup')


// root# grep anonymous /local/jenkins/config.xml
// hudson.model.Hudson.Read:anonymous
// hudson.model.Item.Read:anonymous
// hudson.model.View.Read:anonymous
strategy.add(Jenkins.READ, 'anonymous')
strategy.add(hudson.model.Item.READ, 'anonymous')
strategy.add(hudson.model.View.READ, 'anonymous')




// Agent (Slave < 2.0) - http://javadoc.jenkins-ci.org/jenkins/model/Jenkins.MasterComputer.html
strategy.add(Jenkins.MasterComputer.BUILD, 'admingroup')
strategy.add(Jenkins.MasterComputer.CONFIGURE, 'admingroup')
strategy.add(Jenkins.MasterComputer.CONNECT, 'admingroup')
strategy.add(Jenkins.MasterComputer.CREATE, 'admingroup')
strategy.add(Jenkins.MasterComputer.DELETE, 'admingroup')
strategy.add(Jenkins.MasterComputer.DISCONNECT, 'admingroup')


// Job - http://javadoc.jenkins-ci.org/hudson/model/Item.html
strategy.add(hudson.model.Item.BUILD, 'admingroup')
strategy.add(hudson.model.Item.CANCEL, 'admingroup')
strategy.add(hudson.model.Item.CONFIGURE, 'admingroup')
strategy.add(hudson.model.Item.CREATE, 'admingroup')
strategy.add(hudson.model.Item.DELETE, 'admingroup')
strategy.add(hudson.model.Item.DISCOVER, 'admingroup')
strategy.add(hudson.model.Item.EXTENDED_READ, 'admingroup')
strategy.add(hudson.model.Item.READ, 'admingroup')
strategy.add(hudson.model.Item.WIPEOUT, 'admingroup')
strategy.add(hudson.model.Item.WORKSPACE, 'admingroup')


// Run - http://javadoc.jenkins-ci.org/hudson/model/Run.html
strategy.add(hudson.model.Run.DELETE, 'admingroup')
strategy.add(hudson.model.Run.UPDATE, 'admingroup')
strategy.add(hudson.model.Run.ARTIFACTS, 'admingroup')


// View - http://javadoc.jenkins-ci.org/hudson/model/View.html
strategy.add(hudson.model.View.CONFIGURE, 'admingroup')
strategy.add(hudson.model.View.CREATE, 'admingroup')
strategy.add(hudson.model.View.DELETE, 'admingroup')
strategy.add(hudson.model.View.READ, 'admingroup')


// SCM - http://javadoc.jenkins-ci.org/hudson/model/View.html
strategy.add(hudson.scm.SCM.TAG, 'admingroup')


// // Credentials - https://github.com/jenkinsci/credentials-plugin/blob/master/src/main/java/com/cloudbees/plugins/credentials/CredentialsProvider.java
// strategy.add(CredentialsProvider.CREATE, "my-user")
// strategy.add(CredentialsProvider.UPDATE, "my-user")
// strategy.add(CredentialsProvider.VIEW, "my-user")
// strategy.add(CredentialsProvider.DELETE, "my-user")
// strategy.add(CredentialsProvider.MANAGE_DOMAINS, "my-user")
//
// Plugin Manager http://javadoc.jenkins-ci.org/hudson/PluginManager.html
//strategy.add(hudson.model.Hudson.UPLOAD_PLUGINS, 'admingroup')
strategy.add(hudson.PluginManager.UPLOAD_PLUGINS, 'admingroup')
strategy.add(hudson.PluginManager.CONFIGURE_UPDATECENTER, 'admingroup')
//
instance.setAuthorizationStrategy(strategy)
instance.save()

64 bit inodes on XFS filesystems of more than 1TB - can't run 32 bit Java client

New server build, replicating existing setup, and user cannot run 32 bit Java client:

[root]~# /local/sw/sos/java/bin/java -version
Error: no `server' JVM at `/local/sw/sos/java/jre/lib/i386/server/libjvm.so'.

The old server is fine

Move it out of the /local/sw filesystem and it runs fine


root@notworking# df -h  /local/sw/sos/java/jre/lib/i386/server/libjvm.so
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup01-sw
                      1.1T  3.6G  1.1T   1% /local/sw

root@working# df -h  /local/sw/sos/java/jre/lib/i386/server/libjvm.so
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup01-sw
                      500G  5.6G  495G   2% /local/sw

The filesystem is > 1 Tb, and according to http://www.tcm.phy.cam.ac.uk/sw/inodes64.html, recent versions of XFS use 64 bit inodes for filesystems more than 1 Tb

Reduce the size of the filesystem, and we are good!

Ugh Openstack - project quota usage

Ugh OpenStack

How do I find my OpenStack project's quota usage? Not obvious...

% nova absolute-limits

+--------------------+------+-------+

| Name               | Used | Max   |

+--------------------+------+-------+

| Cores              | 2    | 4     |

| FloatingIps        | 0    | 4     |

| ImageMeta          | -    | 128   |

| Instances          | 2    | 4     |

| Keypairs           | -    | 100   |

| Personality        | -    | 5     |

| Personality Size   | -    | 10240 |

| RAM                | 4096 | 16384 |

| SecurityGroupRules | -    | 20    |

| SecurityGroups     | 1    | 10    |

| Server Meta        | -    | 128   |

| ServerGroupMembers | -    | 10    |

| ServerGroups       | 0    | 10    |

+--------------------+------+-------+

virt-install hangs at boot menu on RHEL 7

Today was yak shaving the virt-install environment from RHEL 6 to RHEL 7

We auto create a custom ISO for virt-install to consume and point to the kickstart server. We use the serial console for everything

On RHEL 6, we can see the BIOS / syslinux boot menu confirming to boot from the ISO

On RHEL 7, we just get a blank screen...

Many hours of determining if the contents of the ISO created on RHEL 6 can work on RHEL 7, and googling just about every combination of virt-install, hang, RHEL 7, CentOS, and RTFM virt-install, I saw a new flag to virt-install on RHEL 7

--boot menu=on,useserial=on
Enable the bios boot menu, and enable sending bios text output over serial console.

And we instantly see the boot menu dialog, and back to normal building!

Leaving this here for anyone else having issues seeing the boot menu over serial

Going to Work

A typical journey to work...

Walking to the Mosman Bay ferry wharf:


Here comes the 8:30 ferry:



View of the City coming around Cremorne Point:


Sydney Icons from Cremorne Point:


Kirribilli:


North Sydney:


The Rocks, coming into Circular Quay:


Coming into the ferry wharf at Circular Quay - Grosvenor Place is the centre building


The bridge from Circular Quay:


Walking up through Herald Square looking at the Tank Stream Fountain:


Work - Level 44 of Grosvenor Place:


View from kitchen - looking out to Cremorne Point - on a lovely Autumn morning:


View from kitchen - looking over The Rocks to North Sydney: