BASH Script On Raspberry Pi To Show Filehandles

Solved an interesting problem on Unix recently that I automated with the shell; which I will demonstrate on Raspberry Pi

QUICK BACKGROUND

I performed a logrotate on a log file which performed the following transformation: 

service.log => service.log.1

which is pretty straight forward, but here was the problem:

After logrotate did its thing, logs were being written to the archived log file service.log.1 and NOT service.log.

This of course defeats to whole purpose of logrotate, so to begin troubleshooting, I needed to answer this question:

What process (or processes) are keeping a handle on that file being logrotated?

MY METHOD

So my first step was to figure out what command I could start with to answer my question.

After a quick google search and testing, I found fuser to be a great spring board.

I began my testing by using the common binary file /usr/sbin/sshd:

root@raspberrypi:~# fuser /usr/sbin/sshd
/usr/sbin/sshd: 406e 776e 846e

Now, according to man 1 fuser, it:

outputs only the PIDs to stdout, everything else is sent to stderr.

Hence, the following will clean up the original output a bit: 

root@raspberrypi:~# fuser /usr/sbin/sshd 2> /dev/null
406 776 846root@raspberrypi:~#

(No newline is printed at the end of the output so the prompt was returned back on the same line.)

Now, if we pipe the output to xargs, it will work and give use the information we want. But not in most user-friendly format:

root@raspberrypi:~# fuser /usr/sbin/sshd 2> /dev/null | xargs -n1 ps -p
PID TTY TIME CMD
406 ? 00:00:00 sshd
PID TTY TIME CMD
776 ? 00:00:00 sshd
PID TTY TIME CMD
846 ? 00:00:00 sshd

Not too pretty.

Reason being, it ran a ps -p on EACH PID.

Instead, we want to run a single ps -p on ALL PIDs at once:

Here’s a way to run ps -p on ALL PIDS at once (-f added to ps command):

root@raspberrypi:~# ps -p $(fuser /usr/sbin/sshd 2> /dev/null | tr -s ' ' | sed -e 's/ /,/g' -e 's/^,//') -f
UID PID PPID C STIME TTY TIME CMD
root 406 1 0 22:30 ? 00:00:00 /usr/sbin/sshd -D
root 776 406 0 22:31 ? 00:00:00 sshd: pi [priv]
pi 846 776 0 22:31 ? 00:00:00 sshd: pi@pts/0

Let’s pause here and make a couple of quick observations:

  • The end result is getting more useful
  • But getting there (the code) is starting to outgrow the shell
  • And, its not easy to re-run this code on ANY filename

Enter…

SCRIPTING

If we “parameterize” the filename AND make it a script, the end result will look something like this:

root@raspberrypi:~/Documents# sh ./list_filehandles.sh /usr/sbin/sshd

Here is the source code for ./list_processhandles.sh:

#!/bin/sh

echo "******************************************"
echo "*"
echo -n "* Parameter passed in: "
echo $1
echo "*"
echo "******************************************"

export MYFILE=$1

ps -p $(fuser $MYFILE 2> /dev/null  | tr -s ' ' | sed -e 's/ /,/g' -e 's/^,//') -f

Now we can fuser ANY file we want to investigate (30 second demo below):

sh ./list_filehandles.sh <filename>

Quickly Create An Ansible Task (Part 1)

Ansible is a great, agentless, automation tool any linux SYSADMIN needs to learn and implement NOW!

Especially if you have at least 1/2 a dozen or more linux boxes.

In this post, I’m assuming you already have Ansible installed and configured (ansible.cfg).

This not a introduction to Ansible, this is specifically for current Ansible users.

We’re going to take the bash command

apt-get install nano

 

And convert it to an Ansible command (there’s actually 3 different kindz!)

BRAINSTORMING

Every Ansible task has a syntax (duh! I know).

But BEFORE we us it, lettuce understand it and visualize it FIRST!!

In our mind’s-eye and with pencil and paper..all one-in-the-same IMHO.

Below is my own visualization of any Ansible command.

This representation is totally arbitrary and is just how I “see it”:

<module_name>
<parameter1>=<value1>
<parameter2>=<value2>
...
<parameterN>=<valueN>

 

Now, it just so happens that Ansible has a module specifically for ‘apt-get’ commands

If we take what Ansible provides, and again, visualize it, we get something like this:

apt
allow_unauthenticated=[no|yes]
autoclean=[no|yes]
autoremove=[no|yes]
...
upgrade=[dist|full|no|safe|yes]

 

Remember, we have not written any Ansible code yet. We are still visualizing and brainstorming. Which is probably the most important part of this entire process anyway.

Next, we will select only that parameters and values we needs for our particular purpose.

 

apt
name=nano
state=absent

 

SUMMARY (So far..)

We took our original bash command:

apt-get install nano

and visualized it according to Ansible’s module page and got:

apt
name=nano
state=absent

Now that we have this visualization, the rest of our mission becomes 1000x easier.

We will take our visualization and transform it into valid Ansible code in our next post.

And because we did most of the cognitive heavy-lifting upfront: brainstorming!

The rest of our work, writing actual Ansible syntax, becomes a lot easier.

 

Exposing Python’s Module Search Path!

I recently bought a Raspberry Pi and started getting my hands dirty with Python (again).

The project I copied was very simple and all it did was make an LED blink.

But one line from the program puzzled me:

import RPi.GPIO as GPIO

So naturally, once I got confused, I started to get curious…the programmer in me asked:

  • What is that line actually doing (beside obviously importing a Python module)?
  • What does the code from that module actually look like?
  • Which directories did Python actually search to find the module?

Knowing this will allow me to dig a little deeper with programming, Python, and Raspberry Pi.

Especially when I start developing my own projects from scratch to build my multi-million dollar solution 🤑 But I digress.

THE CODE

Go a head and copy the following code into path.py:

#!/usr/bin/python

import sys

for p in sys.path:
print p

Then call it from your command line:

root@raspberrypi:~# python path.py
/root
/usr/lib/python2.7
/usr/lib/python2.7/plat-arm-linux-gnueabihf
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages

The above output shows us every single directory Python searched when it hit the line:

import RPi.GPIO as GPIO

NOW, IT GETS EVEN BETTER!

We can take this a step further and see every Python module we have available to us:

root@raspberrypi:~# python path.py | xargs -n1 -I {} find {} -name "*.py"

Now, we can search for the module I was curious about in the beginning:

root@raspberrypi:~# python path.py | xargs -n1 -I {} find {} -name "*.py" | grep RPi.GPIO
/usr/lib/python2.7/dist-packages/RPi/GPIO/__init__.py
/usr/lib/python2.7/dist-packages/RPi/__init__.py
find: ‘/usr/lib/python2.7/lib-old’: No such file or directory
/usr/lib/python2.7/dist-packages/RPi/GPIO/__init__.py
/usr/lib/python2.7/dist-packages/RPi/__init__.py

Not to mention, we have a pretty good troubleshooting technique as well.

QUICK DEMONSTRATION BELOW

Read and Troubleshoot Linux Config Files 10x Faster!

One of the biggest pain-in-rears with being a Linux admin is reading config files.

But not just reading them: TROUBLESHOOTING them.

Config files can get very long, and have a lot of – albeit helpful – screen crowding comments and empty lines.

Comments and empty lines are just something we don’t care about when we are TROUBLESHOOTING.

We just want to read and focus in on what settings are actually being implement and go from there..

One of the things we want to know – and quickly – is:

What is the current configuration of my daemon?

or stated another way:

What settings am I actually implementing right now?

The Script

This script I wrote removes all the “fluff” from a config file and ONLY SHOWS ME WHAT

SETTINGS I’M ACTUALLY IMPLEMENTING 

It removes comments, removes empty lines, and essential compacts the output.

Oh, and there’s a bonus feature I’ll show later in this post…so stick around!

But here is the script:

#!/bin/bash


# This bash script will remove all commented lines or any lines that start
# with a new line character (preceded by 0 or more whitespaces).
# A whitespace in this script is defined as: [ \r\t\f]
#
# This script is great mostly for config files where you want to remove all the "fluff" and see only
# what the config file is implementing. Side note, the bash script just invokes a Perl one-liner.
#
# The file can be passed to it as an argument:
#
# ./hideComments.sh  /etc/ssh/sshd_config
#
# or PIPED to it:
#
# cat /etc/ssh/sshd_config |   ./hideComments.sh


# A whitespace in this script is defined as: [ \r\t\f]
/usr/bin/perl -ne  ' unless ( /^[ \r\t\f]*#|;/ || /^[ \r\t\f]*\n/) {

#color-code common "section bracket line"
# i.e. hc /etc/samba/smb.conf
if ( /^\[/ ){
 print   "\033[0;32m$_\033[0m"
}

else {
 print   "$_"
}


} ' $1

To make this a callable script do the following (assuming you have root privileges):

  1. Save the above in /usr/local/bin/u_hc
  2. chmod 555 /usr/loca/bin/u_hc
  3. chown root:root /usr/local/bin/u_hc

Troubleshooting /etc/ssh/sshd_config

Now, if we look at a default config file for SSHD, it looks like this:

# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

That’s a lot of gunk!

Now again, it’s very useful information.

But if we are TROUBLESHOOTING the daemon, it’s gunk!

If we want to remove ALL of the “fluff” and just read what is ACTUALLY being implemented we run our script as such:

root@DESKTOP-EF1VT0V:~# u_hc /etc/ssh/sshd_config

And the above gets transformed into this:

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes

Troubleshooting /etc/samba/smb.conf (with bonus feature…)

The current version of the script you see above not only:

  • Removes comments (lines beginning with # or ; )
  • Removes empty lines

But it also color codes INI-style syntax

Sometimes with full-blown config files, not only is it hard to see what settings we are enforcing..but what sections these settings are being applied to

For eggsample, take /etc/smb.conf:

# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example
#
# For a step to step guide on installing, configuring and using samba,
# read the Samba-HOWTO-Collection. This may be obtained from:
#  http://www.samba.org/samba/docs/Samba-HOWTO-Collection.pdf
#
# Many working examples of smb.conf files can be found in the
# Samba-Guide which is generated daily and can be downloaded from:
#  http://www.samba.org/samba/docs/Samba-Guide.pdf
#
# Any line which starts with a ; (semi-colon) or a # (hash)
# is a comment and is ignored. In this example we will use a #
# for commentry and a ; for parts of the config file that you
# may wish to enable
#
# NOTE: Whenever you modify this file you should run the command "testparm"
# to check that you have not made any basic syntactic errors.
#
#======================= Global Settings =====================================
[global]

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
   workgroup = MYGROUP

# server string is the equivalent of the NT Description field
   server string = Samba Server

# Server role. Defines in which mode Samba will operate. Possible
# values are "standalone server", "member server", "classic primary
# domain controller", "classic backup domain controller", "active
# directory domain controller".
#
# Most people will want "standalone sever" or "member server".
# Running as "active directory domain controller" will require first
# running "samba-tool domain provision" to wipe databases and create a
# new domain.
   server role = standalone server

# This option is important for security. It allows you to restrict
# connections to machines which are on your local network. The
# following example restricts access to two C class networks and
# the "loopback" interface. For more examples of the syntax see
# the smb.conf man page
;   hosts allow = 192.168.1. 192.168.2. 127.

# Uncomment this if you want a guest account, you must add this to /etc/passwd
# otherwise the user "nobody" is used
;  guest account = pcguest

# this tells Samba to use a separate log file for each machine
# that connects
   log file = /usr/local/samba/var/log.%m

# Put a capping on the size of the log files (in Kb).
   max log size = 50

# Specifies the Kerberos or Active Directory realm the host is part of
;   realm = MY_REALM

# Backend to store user information in. New installations should
# use either tdbsam or ldapsam. smbpasswd is available for backwards
# compatibility. tdbsam requires no further configuration.
;   passdb backend = tdbsam

# Using the following line enables you to customise your configuration
# on a per machine basis. The %m gets replaced with the netbios name
# of the machine that is connecting.
# Note: Consider carefully the location in the configuration file of
#       this line.  The included file is read at that point.
;   include = /usr/local/samba/lib/smb.conf.%m

# Configure Samba to use multiple interfaces
# If you have multiple network interfaces then you must list them
# here. See the man page for details.
;   interfaces = 192.168.12.2/24 192.168.13.2/24

# Where to store roving profiles (only for Win95 and WinNT)
#        %L substitutes for this servers netbios name, %U is username
#        You must uncomment the [Profiles] share below
;   logon path = \\%L\Profiles\%U

# Windows Internet Name Serving Support Section:
# WINS Support - Tells the NMBD component of Samba to enable it's WINS Server
;   wins support = yes

# WINS Server - Tells the NMBD components of Samba to be a WINS Client
#       Note: Samba can be either a WINS Server, or a WINS Client, but NOT both
;   wins server = w.x.y.z

# WINS Proxy - Tells Samba to answer name resolution queries on
# behalf of a non WINS capable client, for this to work there must be
# at least one  WINS Server on the network. The default is NO.
;   wins proxy = yes

# DNS Proxy - tells Samba whether or not to try to resolve NetBIOS names
# via DNS nslookups. The default is NO.
   dns proxy = no

# These scripts are used on a domain controller or stand-alone
# machine to add or delete corresponding unix accounts
;  add user script = /usr/sbin/useradd %u
;  add group script = /usr/sbin/groupadd %g
;  add machine script = /usr/sbin/adduser -n -g machines -c Machine -d /dev/null -s /bin/false %u
;  delete user script = /usr/sbin/userdel %u
;  delete user from group script = /usr/sbin/deluser %u %g
;  delete group script = /usr/sbin/groupdel %g


#============================ Share Definitions ==============================
[homes]
   comment = Home Directories
   browseable = no
   writable = yes

# Un-comment the following and create the netlogon directory for Domain Logons
; [netlogon]
;   comment = Network Logon Service
;   path = /usr/local/samba/lib/netlogon
;   guest ok = yes
;   writable = no
;   share modes = no


# Un-comment the following to provide a specific roving profile share
# the default is to use the user's home directory
;[Profiles]
;    path = /usr/local/samba/profiles
;    browseable = no
;    guest ok = yes


# NOTE: If you have a BSD-style print system there is no need to
# specifically define each individual printer
[printers]
   comment = All Printers
   path = /usr/spool/samba
   browseable = no
# Set public = yes to allow user 'guest account' to print
   guest ok = no
   writable = no
   printable = yes

# This one is useful for people to share files
;[tmp]
;   comment = Temporary file space
;   path = /tmp
;   read only = no
;   public = yes

# A publicly accessible directory, but read only, except for people in
# the "staff" group
;[public]
;   comment = Public Stuff
;   path = /home/samba
;   public = yes
;   writable = no
;   printable = no
;   write list = @staff

# Other examples.
#
# A private printer, usable only by fred. Spool data will be placed in fred's
# home directory. Note that fred must have write access to the spool directory,
# wherever it is.
;[fredsprn]
;   comment = Fred's Printer
;   valid users = fred
;   path = /homes/fred
;   printer = freds_printer
;   public = no
;   writable = no
;   printable = yes

# A private directory, usable only by fred. Note that fred requires write
# access to the directory.
;[fredsdir]
;   comment = Fred's Service
;   path = /usr/somewhere/private
;   valid users = fred
;   public = no
;   writable = yes
;   printable = no

# a service which has a different directory for each machine that connects
# this allows you to tailor configurations to incoming machines. You could
# also use the %U option to tailor it by user name.
# The %m gets replaced with the machine name that is connecting.
;[pchome]
;  comment = PC Directories
;  path = /usr/pc/%m
;  public = no
;  writable = yes

# A publicly accessible directory, read/write to all users. Note that all files
# created in the directory by users will be owned by the default user, so
# any user with access can delete any other user's files. Obviously this
# directory must be writable by the default user. Another user could of course
# be specified, in which case all files would be owned by that user instead.
;[public]
;   path = /usr/somewhere/else/public
;   public = yes
;   only guest = yes
;   writable = yes
;   printable = no

# The following two entries demonstrate how to share a directory so that two
# users can place files there that will be owned by the specific users. In this
# setup, the directory should be writable by both users and should have the
# sticky bit set on it to prevent abuse. Obviously this could be extended to
# as many users as required.
;[myshare]
;   comment = Mary's and Fred's stuff
;   path = /usr/somewhere/shared
;   valid users = mary fred
;   public = no
;   writable = yes
;   printable = no
;   create mask = 0765

WOW!

That’s insanely long!

Once again, our script will save us.

The screenshot below shows us the “compressed” version and the color-coded INI-style syntax as well!

Untitled

Now that’s a 🌎 of difference.

Same config settings, but A LOT easier and faster to read with our script.

Apply this script to some of your config files and let me know if it helps you TROUBLESHOOT better 😀

Learning Linux Bash with tar command

In this post, I’ll use the tar command to reveal some secrets of the CLI-Universe.

You may have seen the following before but had no clue what the heck it meant:

tar -cvzf /tmp/backup.bzip.tar /tmp/example.txt

Not only am I going to show you what this means (give you a fish), but I’m going to teach you how to actually READ IT (teach you how to fish).

KEEP THIS MIND 🤔💭:

The beauty about this process I’m going to show you is that it can be done:

  • Mentally at the speed of thought (once you’ve done this a few times)
  • Or physically on good ‘ol fashion pencil and paper
# But first, lettuce create our example test file
echo "tar me baby!" > /tmp/example.txt

Kay, here we go..

1.) Expand Any Short-hand Notation

The first thing to notice is we have some short-hand notation to expand.

We re-write:

tar -cvzf /tmp/backup.bzip.tar /tmp/example.txt

as:

tar -c -v -z -f /tmp/backup.bzip.tar /tmp/example.txt

2.) Convert to a Multi-line Command

Now, instead of having one long disgusting line 🤮, we take this command

tar -c -v -z -f /tmp/backup.bzip.tar /tmp/example.txt

and re-write it as:

tar \ [press enter]
> -c \ [press enter]
> -v \ [press enter]
> -z \ [press enter]
> -f /tmp/backup.bzip.tar \ [press enter]
> /tmp/example.txt [press enter]

Here’s a short video demonstrating the above transformations:

3.) Expand to Long-form Notation

If we came back to this “code” 6 months later, we would probably need the man page –  🤮 again – to remember what each parameter did.

Especially if we don’t use tar everyday.

So, as a best practice, we should convert the parameters to their long-form:

tar \ [press enter]
> --create \ [press enter]
> --verbose \ [press enter]
> --gzip \ [press enter]
> --file /tmp/backup.bzip.tar \ [press enter]
> /tmp/example.txt [press enter]

4.) Put Back to One Line

For contrast, if we put this back to one-line, we can see that:

tar -cvzf /tmp/backup.bzip.tar /tmp/example.txt

is equivalent to:

tar  --create --verbose --gzip --file /tmp/backup.bzip.tar  /tmp/example.txt

So What ?!…

This technique of deconstructing can be VERY helpful when you have commands to read.

This technique is also very helpful if you are trying to construct your own commands.

MY RECOMMENDATION FOR WRITING COMPLEX COMMANDS:

  1. Start with the long-form notations of the command
  2. Be very explicit, no short-hand or short-form notation
  3. Once you understand what you wrote, THEN convert it to short-form
  4. Then convert to short-hand if applicable
  5. Lastly, give the short-hand version to the new guy to read…for learning purposes of course 😉

MY RECOMMENDATION FOR READING COMPLEX COMMANDS:

  1. Expand any short-hand notations to short-form notations (-cv to -c -v)
  2. Transform to a multi-line command
  3. Expand any short-form notations to long-form notations (-c to –create)
  4. Put back to multi-line for contrast

Was this article helpful?

Comment Below