Lazy Automation

[Michael]

Lets face it we are all a little bit lazy and why do something the hard way when you can get a computer to do it for you. This topic is to share tasks you’ve automated to make your life easier or more efficient, be it in Bash, Python, C, PowerShell, Excel, hell even Redstone, if you’ve automated it I want to see it.

mp3 conversion

I made this script a while ago to convert my music collection to mp3. It takes three arguments either wav, flac, or ogg and converts all files in the current directory to mp3.

#!/bin/bash

# loop through files in the currently active directory
for f in *.$1
    do
        # case statement for arguments (you can convert wav, flac, and ogg to mp3)
        case $1 in
            wav)
                # converts wav files to mp3 using lame
                lame -b320 "$f" "${f%.*}".mp3
                ;;
            flac)
                # converts flac files to mp3 using flac/lame
                flac -cd "$f" | lame -b320 - "${f%.*}".mp3
                ;;
            ogg)
                # converts ogg to mp3 using oggdec/lame
                oggdec "$f" -o - | lame -b320 - "${f%.*}".mp3
                ;;
            *)
                echo "Usage: $0 {wav|flac|ogg}"
        esac
        shopt -s nullglob
        # check for mp3 files in current directory
        if [[ -n $(echo *.mp3) ]]; then
            # check if mp3 directory exists
            if [ ! -d "mp3" ]; then
                # creates mp3 directory if it doesn't already exist
                mkdir mp3
            fi
            # move all mp3 files to the mp3 directory
            mv *.mp3 mp3/
        fi
    done

FLAC conversion

After a while of using mp3 for my music I decided to switch to FLAC so adapted the script above to convert wav files to flac:

#!/bin/bash

# loop through files in currently active directory
for f in *.wav
    do
        # converts wav files to flac using sox
        sox "$f" "${f%.*}".flac
        shopt -s nullglob
        #check for flac files in current directory
        if [[ -n $(echo *.flac) ]]; then
            # check if flac directory exists
            if [ ! -d "flac" ]; then
                # creates flac direcotry if it doesn't already exist
                mkdir flac
            fi
            # move all flac files to the flac directory
            mv *.flac flac/
        fi
    done

LFS Chroot

When I built Linux from Stretch I often needed to detach my usb flash drive to test it but also center the chroot again later so I made this. it takes two arguments m and u, the first mounts everything and enters the chroot and the second unmounts and cleans everything up.

#!/bin/bash

export LFS=/mnt/lfs

case $1 in
    m)
        mount -v -t ext4 /dev/sde1 $LFS
        mount -v -t ext4 /dev/sde2 $LFS/home

        mount -v --bind /dev $LFS/dev

        mount -vt devpts devpts $LFS/dev/pts -o gid=5,mode=620
        mount -vt proc proc $LFS/proc
        mount -vt sysfs sysfs $LFS/sys
        mount -vt tmpfs tmpfs $LFS/run

        if [ -h $LFS/dev/shm ]; then
          mkdir -pv $LFS/$(readlink $LFS/dev/shm)
        fi

        chroot "$LFS" /usr/bin/env -i              \
            HOME=/root TERM="$TERM" PS1='\u:\w\$ ' \
            PATH=/bin:/usr/bin:/sbin:/usr/sbin     \
            /bin/bash --login
    ;;
    u)
        umount -v $LFS/dev/pts
        umount -v $LFS/dev
        umount -v $LFS/run
        umount -v $LFS/proc
        umount -v $LFS/sys

        umount -v $LFS/home
        umount -v $LFS
        ;;
    *)
        echo "Usage: $0 {m|u}"
esac

I also made a script to automate the creation of a bootable Windows flash drive which I posted in more detail here:

Creating Bootable Windows Flash Drives on Linux

I often find myself in a situation where I need to create bootable Windows USBs which I have found to be a very daunting task on Linux as most of the tools out there tend to be unreliable. My solution to this problem is to do it manually and write script to automate it so that its consistent and repeatable. The Theory and Test build These are the general steps that required to successfully create a Windows USB: Create partition table on drive as type 7 and bootable Format drive as NTFS Write …

Thats it for now, I’ll post new stuff later when I have more to show.

[Dje4321 (The Packet Wrangler)]

A game i like to play often is cataclysm-dda but i wanted to keep up with the latest development versions so i make a script that will check for updates and compile 2 versions of the game. One with tile support and one without

#!/bin/bash

UPDATES=$(git pull) #check for updates and download them. Store the results to check to see if we got updates
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ) #gets the current directory and saves the output

RESPONSE="N" #default response used when already upto date
INSTALL_DIR="install/"
MAKE_FLAGS="-j1"

shopt -s extglob

if [[ $UPDATES = "Already up to date." ]] #Checks to see if the latest version is running and if true asks if you want to go ahead and compile
then
        echo "Already upto date."
        echo "Continue Building (y/N)?"
        read RESPONSE
        if [[ ${RESPONSE,,} != "y" ]]
        then
                exit
        fi

fi

if [ -e output.log ] #checks if the log already exsists and if true it removes it
then
        rm output.log
fi

make clean |& tee -a output.log #prepare the source folder for compilation

echo "#############################" |& tee -a output.log
echo "Compiling with tiles build" |& tee -a output.log
echo "#############################" |& tee -a output.log
make $MAKE_FLAGS RELEASE=1 LTO=1 TILES=1 SOUND=1 LUA=1 CLANG=1 CCACHE=1 |& tee -a output.log #compiles a version of cataclysm with tiles

echo "#############################" |& tee -a output.log
echo "Compiling default build" |& tee -a output.log
echo "#############################" |& tee -a output.log
make $MAKE_FLAGS RELEASE=1 LTO=1 LUA=1 CLANG=1 CCACHE=1 |& tee -a output.log #compiles a non tiles build of cataclysm

echo "#############################" |& tee -a output.log
echo "Installing cataclysm DDA" |& tee -a output.log
echo "############################" |& tee -a output.log

if [ ! -d $DIR/$INSTALL_DIR ] #check to see if the install folder exsists and if it does not it creates it
then
        mkdir $DIR/$INSTALL_DIR
fi

rm -rf $DIR/$INSTALL_DIR/!(save|config|templates) #remove the old version of the game
cp -r $DIR/data/ $DIR/lua/ $DIR/gfx/ $DIR/cataclysm $DIR/cataclysm-tiles $INSTALL_DIR #copys over the latest bulid to $INSTALL_DIR

Dont judge too hard. is the first real script ive made in a LONG time

[Michael]

Some more scripts I’ve made because why not.

BTRFS snaphots

Quick and dirty script I made to automatically take BTRFS snapshots of my root and /home partitions. I run this daily on a cronjob:

#!/bin/bash

snapshot_dir="/mnt/snapshots"
date=$(date +"%Y-%m-%d--%H:%M:%S")

echo -e "\e[33mCreating snapshots\033[0m"
btrfs subvol snapshot / $snapshot_dir/root/$date
btrfs subvol snapshot /home $snapshot_dir/home/$date

echo -e "\e[33m$snapshot_dir/root\033[0m"
btrfs subvol list -o $snapshot_dir/root
echo -e "\e[33m$snapshot_dir/home\033[0m"
btrfs subvol list -o $snapshot_dir/home

pfSense config backup

pfSense only allows you to automatically backup your router config if you have a gold subscription, so I made this script that logins into the admin panel and retrieves the latest config and downloads it to my desktop using wget.

#!/bin/bash

backup="/mnt/storage/storage/Network/pfSense"
$password="password"
cd /tmp

wget -qO- --keep-session-cookies --save-cookies cookies.txt \
  --no-check-certificate https://10.0.1.1/diag_backup.php \
  | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > csrf.txt

wget -qO- --keep-session-cookies --load-cookies cookies.txt \
--save-cookies cookies.txt --no-check-certificate \
--post-data "login=Login&usernamefld=admin&passwordfld=$password&__csrf_magic=$(cat csrf.txt)" \
https://10.0.1.1/diag_backup.php  | grep "name='__csrf_magic'" \
| sed 's/.*value="\(.*\)".*/\1/' > csrf2.txt

wget --keep-session-cookies --load-cookies cookies.txt --no-check-certificate \
  --post-data "download=download&donotbackuprrd=yes&__csrf_magic=$(head -n 1 csrf2.txt)" \
  https://10.0.1.1/diag_backup.php -O config-router-`date +%Y%m%d%H%M%S`.xml

mv config-router* $backup

Mount NFS partition

A very long time ago I had issues on ubuntu automatically mounting my NFS connection from my server on bootup using fstab. so I made this script to either run on startup in my rc.local or manaully if I needed to mount it after the system has started.

#!/bin/bash

address="192.168.0.6"
mount_point="/mnt/nfs-storage"

if [[ $EUID -ne 0 ]]; then
    echo "This script must be run as root"
    exit 1
fi

case $1 in
    mount)
        if mountpoint -q $mount_point; then
            echo "$mount_point is active"
        else
            timeout 5 mount $address:/mnt/storage $mount_point
        fi
        ;;
    umount)
        if mountpoint -q $mount_point; then
            umount -f -l $mount_point
        else
            echo "$mount_point is not mounted"
        fi
        ;;
    *)
        echo "Usage $0 {mount|umount}"
esac

[Michael]

I got tired of manually compiling and packaging my LineageOS builds so I made this script to automate it. nothing too fancy but it gets the job done

#!/bin/bash

CODENAME="h850" # device codename
CERTS_DIR="$HOME/android-certs" # certs directory for signing package
PACKAGE_DIR="$HOME/lineage-package" # package install directory
RELEASE_TOOLS="build/tools/releasetools" # android release tools
ENVFILE="build/envsetup.sh" # environment file
DATE=$(date +"%Y-%m-%d--%H:%M:%S")

source $ENVFILE # setup environment
export USE_CCACHE=1 # enable ccache
export CCACHE_COMPRESS=1 # compress ccache

breakfast $CODENAME # prepare environment for target device
rm -rf out/dist/lineage_$CODENAME* # remove old target files
time mka target-files-package dist # compile and package target files

# generate signed target files
./$RELEASE_TOOLS/sign_target_files_apks -o -d $CERTS_DIR \
	out/dist/lineage_${CODENAME}-target_files-*.zip \
	signed-target_files.zip

# generate install package
./$RELEASE_TOOLS/ota_from_target_files -k $CERTS_DIR/releasekey \
	--block --backup=true \
	signed-target_files.zip \
	signed-ota_update-$DATE.zip

mv signed-target_files.zip $PACKAGE_DIR/signed-target_files-$DATE.zip # move signed target files to package directory
mv signed-ota_update-* $PACKAGE_DIR # move install package to package directory

[Michael]

When using the script above I noticed that my compile times were significantly longer than expected as I’m a dummy and I forgot to add the ccache variables. I’ve edited the script to include them.

[Michael]

Very simple script I made that pops up an i3 message bar after a specific amount of time.

#!/bin/bash

TIME=$1
MESSAGE=$2

if [[ -z $1 ]]; then TIME=1s; fi
if [[ -z $2 ]]; then MESSAGE="Your timer is ready"; fi

sleep $TIME && exec i3-nagbar -t warning -m "$MESSAGE"

This allows me to set a timer when its too late to otherwise set it on my phone and no more will I burn pizza

I can also be extra lazy and further automate it by setting an i3 hotkey for instant Pizza notification:

bindsym $mod+F10 exec ~/bin/timer 15m "Your Pizza is ready"

[Dje4321 (The Packet Wrangler)]

I made a neat little python program that will check for errors. Is mostly due to the fact that im too lazy to just open up a terminal to find any errors myself.

Right now it only checks for errors on systemd and disk usage but when i think of more things i would like checked than ill add that in too

Added in support for checking entropy and whether or not there is anything in trash

dje4321/LinuxStatus

[Michael]

When installing Fedora 28 I forgot to backup my LineageOS container so I’ve made a build to automate the creation of a new container and a custom init script that sets up the build environment as much as possible. This does a couple of things:

1. Installs all the required packages needed to compile LineageOS
2. Creates a user with a pre-defined .bashrc and .profile for the build environment
3. Downloads and install the Android repo and platform-tools
4. Create and syncs the repository ready to build LineageOS

michaellindman/docker-builds

I also included my previous docker builds for flarum and tiddilywiki as well because why not.

[Michael]

Small script I made that will rotate the display and touchscreen input device on my laptop so that they are the correct orientation. The accelerometer doesn’t auto rotate in i3 like it does in gnome so I’ve setup a mode that calls this script and using the arrow keys to rotate it in whatever orientation I want it

#!/bin/bash

OUTPUT="eDP-1" # laptop display
DEV="ELAN0732:00 04F3:2B45" # touchscreen input device
xrandr --output $OUTPUT --rotate $1 # rotate display

case $1 in
    left)
        xinput set-prop "$DEV" "Coordinate Transformation Matrix" 0 -1 1 1 0 0 0 0 1
        ;;
    right)
        xinput set-prop "$DEV" "Coordinate Transformation Matrix" 0 1 0 -1 0 1 0 0 1
        ;;
    normal)
        xinput set-prop "$DEV" "Coordinate Transformation Matrix" 1 0 0 0 1 0 0 0 1
        ;;
    inverted)
        xinput set-prop "$DEV" "Coordinate Transformation Matrix" -1 0 1 0 -1 1 0 0 1
        ;;
    *)
        echo "Usage: $0 {left|right|normal|inverted}"
esac

[Michael]

Simple python script I made to change the brightness of my displays using xrandr by increments of .25 depending on which workspace is active (I.E which monitor has focus).

#!/usr/bin/env python3

import os, sys, string, subprocess

def main():
    cmd = "xrandr --verbose | grep {} -A 5 | grep Brightness | cut -f 2 -d ' '".format(display())
    brightness = float(subprocess.getoutput(cmd))

    if sys.argv[1] == "+" and brightness < 1:
        brightness += .25
    elif sys.argv[1] == "-" and brightness > 0:
        brightness -= .25
    
    os.system("xrandr --output {} --brightness {}".format(display(), brightness))

def display():
    cmd = "i3-msg -t get_workspaces | jq '.[] | select(.focused==true).name'"
    cmd = subprocess.getoutput(cmd).replace("\"", "")

    if cmd == "8: ":
        return "DP-1"
    return "DP-2"

if len(sys.argv) <= 1:
    print("Usage: {} [+|-]".format(sys.argv[0]))
    sys.exit(1)
else:
    main()

and in i3 i just set the bindings as

bindsym $mod+Shift+F12 exec ~/bin/brightness.py +
bindsym $mod+Shift+F11 exec ~/bin/brightness.py -

so I can now change the brightness levels of my displays on the fly without having to use their horrible OSDs to do it

[Michael]

One of its main issues of the script above is it relies on me never changing the workspace on my second monitor, while this doesn’t happen often in an event that it does it’ll change the brightness of my primary display even though the second has focus, so to fix this i made a couple of improvements.

First I decided to get the location of my mouse and using that extrapolate which display has focused using:

xdotool getmouselocation --shell | head -n -3 | sed 's/[^0-9]*//g'

At first I was going to try and calculate between the x and y which has focus until I realized that all that really matters is that the x of the mouse is below the width of my left most display since the resolutions are combined into Screen 0 so could just be lazy and do a check such as:

if x < 2560:
    return "DP-1"
return "DP-2"

with 2560 being the width of my display and x the mouse coordinates on that axis.

Here is the new script with the changes is below:

#!/usr/bin/env python3

import os, sys, string, subprocess

def main():
    cmd = "xrandr --verbose | grep {} -A 5 | grep Brightness | cut -f 2 -d ' '".format(display())
    brightness = float(subprocess.getoutput(cmd))

    if sys.argv[1] == "+" and brightness < 1:
        brightness += .25
    elif sys.argv[1] == "-" and brightness > 0:
        brightness -= .25
    os.system("xrandr --output {} --brightness {}".format(display(), brightness))

def display():
    cmd = "xdotool getmouselocation --shell | head -n -3 | sed 's/[^0-9]*//g'"
    cmd = subprocess.getoutput(cmd)

    if int(cmd) < 2560:
        return "DP-1"
    return "DP-2"

if len(sys.argv) <= 1:
    print("Usage: {} [+|-]".format(sys.argv[0]))
    sys.exit(1)
else:
    main()

I plan to improve it further by pulling the display id from xrandr based on the coordinates so nothing is hard-coded but that is for another time and this works well enough for my needs now and is vast improvement over my previous version with the added benefit that this now works without requiring the use of i3.

[Michael]

kinda simple one but mother wanted me to print 34 pdf files and when I asked Windows to print them from the file manager it told me i had to open each one and print them individually so transferred them over to Linux and made this small shell script to do it.

#!/bin/sh

for f in *.pdf; do
    cat "$f" | lp -o scaling=//100// -oColorModel=KGray
done

Probably could’ve found a way todo it on Windows too but I’m far less familiar with batch scripting and this way still saves me a lot of time and effort.

[Michael]

@Michael

looking around I can totally do it in the API with posts using curl and could probably write something within a day that’ll work but i’d be a bit hacky.

@tsk Its very rough draft but this should do it. It grabs the user list based from the tier level/group you specify inside the api section and checks if the user is inside the group, if not it will send a POST API request to the forum and assign the group to that user. The script is in two parts, first handles the API requests (api.py).

import requests, json

Api_Key = "" # your discourse api key
groupName = "NSFW" # name of the group
tier_level = 3
url = "https://forum.0cd.xyz"

def _url(path):
    return url + path

def get_group():
    return requests.get(_url('/g/{}.json').format(groupName))

def get_group_members():
    return requests.get(_url('/g/{}/members.json').format(groupName))

def get_tl_members():
    return requests.get(_url('/g/trust_level_{}/members.json').format(tier_level))

def add_users(users, gid):
    headers = {'Api-Key': Api_Key, 'content-type':'application/json'}
    return requests.put(_url("/g/{}/members.json").format(gid), data=json.dumps({'usernames': ','.join(users)}), headers=headers)

and the second the main logic of the program

#!/usr/bin/env python3

import sys, requests, json, api

def main():
    try:
        resp = api.get_tl_members()
        gid = api.get_group()
        if resp.status_code != 200 or gid.status_code != 200:
            raise ApiError('Cannot fetch data: tl: {} group: {}'.format(resp.status_code, gid.status_code))
        member = []
        for users in resp.json()['members']:
            if users['username'] not in group():
                member.append(users['username'])
        api.add_users(member, gid.json()['group']['id'])
    except(ApiError, requests.exceptions.ConnectionError) as e:
        print(e)
        sys.exit(0)

def group():
    try:
        resp = api.get_group_members()
        if resp.status_code != 200:
            raise ApiError('Cannot fetch data: {}'.format(resp.status_code))
        member = []
        for members in resp.json()['members']:
            member.append(members['username'])
        return member
    except(ApiError, requests.exceptions.ConnectionError) as e:
        print(e)
        sys.exit(0)        

class ApiError(Exception): pass

main()

It should work mostly fine except there’s an issue in that the proper way handle the request is broken in Discourse so I’m brute forcing it by iterating over the users and sending a separate API request for each user, this has a downside in that Discourse has a request limit so adding lots of users at once will kill the API requests., haven’t done extensive testing because the only way todo that is in production but tested code works outside of the main for loop and it works fine.

---

fixed most of the issues above. it now makes a single API request with a list of usernames so no chance of hitting the requests limit and its a lot more efficient. old way was to add the users from their page on the admin panel one by one but now it pushes a list using the proper Add user call to the groups API.

[tsk (Legend)]

Massive thanks mate

[Michael]

no problem, was fun to make. if you end up use it and have any issues let me know.

[Michael]

Had todo another one of these large prints again and got complained at because they needed to be in reverse order and double-sided, fixed it up and included a watch for printer queue. thankfully zsh makes it super easy to reverse the order of the file list using the On glob qualifier.

#!/bin/zsh

for f in *.pdf(On); do
    cat "$f" | lp -o scaling=//100// -o ColorModel=KGray -o sides=two-sided-long-edge
done

watch -n1 lpstat

[dinscurge]

lel did this with my media server but it actually paid off now i just hit go(to save power and shit by actually shutting it off at night/when go to work and whatever)

[Michael]

Was bored and decided to rewrite my xrandr brightness control script in golang, it effectively does exactly the same thing as the python version but its now its all fancy with compiled code.

package main

import (
	"fmt"
	"log"
	"os"
	"os/exec"
	"strconv"
	"strings"
)

func main() {
	if len(os.Args) <= 1 {
		fmt.Printf("Usage: %s [+|-]", os.Args[0])
		return
	}
	brightness()
}

func brightness() {
	cmd := fmt.Sprintf("xrandr --verbose | grep %s -A 5 | grep Brightness | cut -f 2 -d ' '", display())
	brightness, err := exec.Command("sh", "-c", cmd).Output()
	if err != nil {
		log.Fatal(err)
	}
	f, err := strconv.ParseFloat(strings.TrimSpace(string(brightness)), 64)
	if err != nil {
		log.Fatal(err)
	}
	switch {
	case os.Args[1] == "+" && f < 1:
		f += .25
	case os.Args[1] == "-" && f > 0:
		f -= .25
	}
	exe := fmt.Sprintf("xrandr --output %s --brightness %s", display(), strconv.FormatFloat(f, 'f', 2, 64))
	exec.Command("sh", "-c", exe).Output()
}

func display() string {
	cmd, err := exec.Command("xdotool", "getmouselocation", "--shell").CombinedOutput()
	if err != nil {
		log.Fatal(err)
	}
	lines := strings.SplitAfter(string(cmd), "\n")
	i, err := strconv.Atoi(strings.TrimSpace(lines[0][len(lines)-3:]))
	if err != nil {
		log.Fatal(err)
	}
	switch {
	case i < 1920:
		return "HDMI-1"
	case i < 4480:
		return "DP-1"
	default:
		return "DP-2"
	}
}

[Michael]

So my mother got some Philips Hue lights for Christmas and messing around with them there 100% local and have an API that can interface directly with the hub. A couple of minutes and dirty shell functions in my .zshrc later I can turn off/on and the brightness of the light in my room from the terminal.

# lights

lights() {
    curl \
        -X PUT \
        -H "Content-Type: application/json" \
        -d "{\"on\": $1}" \
        "http://10.0.2.24/api/YHp15uuWjnbOt8cjRJEnUY-Tf86oMUoL5BpuHypg/lights/4/state"
}

light_bri() {
    curl \
        -X PUT \
        -H "Content-Type: application/json" \
        -d "{
                \"on\": true,
                \"bri\": $1
            }" \
        "http://10.0.2.24/api/YHp15uuWjnbOt8cjRJEnUY-Tf86oMUoL5BpuHypg/lights/4/state"
}

and with the addition of some i3 hotkeys…

bindsym $mod+Shift+h mode "lights"

mode "lights" {
    bindsym o exec zsh -i --login -c "lights true"
    bindsym f exec zsh -i --login -c "lights false"
    bindsym F12 exec zsh -i --login -c "light_bri 254"
    bindsym F11 exec zsh -i --login -c "light_bri 150"

    bindsym Return mode "default"
    bindsym Escape mode "default"
}

I can now turn on/off my lights using only my keyboard without even having to get up