The ANSI escape code for a red color is \033[31m. If we prepend that to a string that we print out to the terminal, the string comes out red. This works regardless of the programming language. Here’s an example in Python:

print("\033[31mthis text is red")

We can split out the ANSI escape code to make it easier to see the two parts:

red_ansi_code = "\033[31m"
message = "this text is red"
print(red_ansi_code + message)

Both code snippets above produce the following output:

this text is red

We can mix colors too. Here is an example with green and blue as well as red. We use the escape code \033[0m to terminate the colorization of text after each string.

print("\033[31mred text\033[0m \033[32mgreen text\033[0m \033[34mblue text\033[0m")

We can split things out once again to make it more legible:

red_ansi_code = "\033[31m"
green_ansi_code = "\033[32m"
blue_ansi_code = "\033[34m"
terminator = "\033[0m"

print(
    red_ansi_code
    + "red text "
    + terminator
    + green_ansi_code
    + "green text "
    + terminator
    + blue_ansi_code
    + "blue text"
    + terminator
)

The two snippets above print:

red text green text blue text

Libraries exist to make this easier since ANSI codes look like nonsense among normal text. For example, there is a termcolor library for Python that lets you wrap text and specify a color.

# import the colored() function from the termcolor library
from termcolor import colored

print(
    colored("red text", "red"),
    colored("green text", "green"),
    colored("blue text", "blue"),
)

That code has the same output as before:

red text green text blue text

The colored function from the termcolor library makes it much cleaner to colorize terminal output.

There are many ANSI codes for colors. Here’s a few examples of different colors:

There are so many more ANSI escape codes to change the color of text, the background color, and even to underline and bold text.

It’s useful to know the general look of ANSI escape codes because they may show up in other places. From personal experience, I’ve seen these codes being printed out along with machine logs and it just looked like gibberish. Now I know that all that gibberish was actually ANSI escape codes that weren’t being rendered. There was code somewhere that was printing colorized output without checking if it was running in a terminal or not.

Here’s an example of raw ANSI codes being printed out from a Rust build with warnings. Take this Rust code:

fn main() {
    let year = 2026;
    println!("hello world");
}

If we try to build it with cargo, we get a warning due to the unused year variable. We can build it with the --color=always option to force colorization. Then we can write the warning to build.log and print out build.log with cat -v so that it reveals raw ANSI codes:

$ cargo build --color=always &> build.log
$ cat -v build.log

^[[1m^[[33mwarning^[[0m^[[1m: unused variable: `year`^[[0m
 ^[[1m^[[94m--> ^[[0msrc/main.rs:2:9
  ^[[1m^[[94m|^[[0m
^[[1m^[[94m2^[[0m ^[[1m^[[94m|^[[0m     let year = 2026;
  ^[[1m^[[94m|^[[0m         ^[[1m^[[33m^^^^^[[0m ^[[1m^[[33mhelp: if this is intentional, prefix it with an underscore: `_year`^[[0m
  ^[[1m^[[94m|^[[0m
  ^[[1m^[[94m= ^[[0m^[[1mnote^[[0m: `#[warn(unused_variables)]` (part of `#[warn(unused)]`) on by default

^[[1m^[[33mwarning^[[0m: `ansidemo` (bin "ansidemo") generated 1 warning (run `cargo fix --bin "ansidemo" -p ansidemo` to apply 1 suggestion)
^[[1m^[[92m    Finished^[[0m `dev` profile [unoptimized + debuginfo] target(s) in 0.10s

Try running both cat build.log and cat -v build.log to see the colorized and non-colorized (with raw ANSI codes) outputs side by side.

import sys

if sys.stdout.isatty():
    print("This program is running in a terminal!")
else:
    print("This program is running somewhere else, like a CI/CD pipeline")

Running the program outputs the following:

$ python3 tty.py
This program is running in a terminal!

But if we pipe the program into some other tool, like cat, it is no longer running in a terminal and prints the other message:

$ python3 tty.py | cat
This program is running somewhere else, like a CI/CD pipeline

This is why some colorized output from code is not colorized in CI/CD pipelines. The code itself checks if it’s being run in a terminal, and if not, it doesn’t display colors that would result in junk being displayed due to the nature of ANSI escape codes used in colorization. I always thought that it was the CI/CD systems that suppressed color output but that’s not necessarily the case.

This capability is not unique to Python. Other languages support this too.

In JavaScript:

if (process.stdout.isTTY) {
  console.log("This program is running in a terminal!");
} else {
  console.log("This program is running somewhere else, like a CI/CD pipeline");
}

In Ruby:

if $stdout.tty?
  puts "This program is running in a terminal!"
else
  puts "This program is running somewhere else, like a CI/CD pipeline"
end

In Bash:

if [ -t 1 ]; then
  echo "This program is running in a terminal!"
else
  echo "This program is running somewhere else, like a CI/CD pipeline"
fi

Go doesn’t have a built-in method to detect if code is being run inside a terminal, so we need to import the x/term package:

package main

import (
	"fmt"
	"os"

	"golang.org/x/term"
)

func main() {
	if term.IsTerminal(int(os.Stdout.Fd())) {
		fmt.Println("This program is running in a terminal!")
	} else {
		fmt.Println("This program is running somewhere else, like a CI/CD pipeline")
	}
}

agg recording.cast recording.gif

However, if you run agg with the default settings the resulting GIF won’t be high quality and it’ll look a little fuzzy.

To get a high quality GIF you need to specify a big font size with the --font-size option:

agg --font-size 64 recording.cast recording.gif

Here are two GIFs you can compare. The top one was created with the default agg settings. The bottom one was created with --font-size 64. Depending on your display, you may need to zoom in a bit to see the difference:

It’s a noticeable difference. The trade-off here is that the higher quality GIF will be a bigger filesize, so be mindful of that. Try experimenting with different font sizes.

I was expecting there to be some sort of --size or --quality option, but you just need to increase the font to get higher quality GIFs with agg.

References

• https://docs.asciinema.org/manual/agg/
• https://docs.asciinema.org/manual/agg/installation/
• https://docs.asciinema.org/manual/agg/usage/

I tried this on macOS but it should work on Linux too.

For example, when I start up a new terminal window/tab I see the following message:

After I create the .hushlogin file in my home directory, the login message goes away. First I created the file:

touch ~/.hushlogin

Then I opened a new terminal window to verify that the message no longer shows up:

References

• https://stackoverflow.com/questions/15769615/remove-last-login-message-for-new-tabs-in-terminal
• https://www.cyberciti.biz/howto/turn-off-the-login-banner-in-linux-unix-with-hushlogin-file/

To clone a specific branch of a git repository without cloning all other branches, use the following command formula:

git clone --single-branch --branch <branch_name> <repo_URL.git>

For example, if you want to clone the release-1.28 branch of the Kubernetes GitHub repository, run:

git clone --single-branch --branch release-1.28 https://github.com/kubernetes/kubernetes.git

Cloning the Latest Commit of a Specific Branch

If you only want to clone the latest commit of a specific branch (which results in a faster and smaller cloning operation) use --depth 1. The command formula looks like this:

git clone --single-branch --branch <branch_name> --depth 1 <repo_URL.git>

And here is another example using the release-1.28 branch of the Kubernetes GitHub repository:

git clone --single-branch --branch release-1.28 --depth 1 https://github.com/kubernetes/kubernetes.git

References

• https://www.freecodecamp.org/news/git-clone-branch-how-to-clone-a-specific-branch/
• https://git-scm.com/docs/git-clone

Delete All Stacks in the Current Project

To delete all Pulumi stacks in the current Pulumi project you can run the following command:

pulumi stack ls | tail -n +2 | tr -d "*" | awk '{print $1}' | while read -r stack; do pulumi stack rm -y "$stack"; done;

Delete All Stacks Across All Projects

Warning:

It should go without saying but, be careful when doing this.

To delete all Pulumi stacks across all Pulumi projects we need to use pulumi stack ls -a instead of pulumi stack ls. So the full command is:

pulumi stack ls -a | tail -n +2 | tr -d "*" | awk '{print $1}' | while read -r stack; do pulumi stack rm -y "$stack"; done;

Command Breakdown

List pulumi stacks (use -a option for all stacks across all projects):

pulumi stack ls

Start at the second line of the previous output:

tail -n +2

Delete all occurrences of *. There is a * character next to the currently selected stack and we need to remove this:

tr -d "*"

Print only the first column:

awk '{print $1}'

This is a loop in one-liner format. It reads the previous output line by line and assigns each line to a string variable called stack, then runs the command pulumi stack rm -y on each stack.

while read -r stack; do pulumi stack rm -y "$stack"; done;

References

• https://stackoverflow.com/questions/7318497/omitting-the-first-line-from-any-linux-command-output
• https://www.pulumi.com/docs/iac/cli/commands/pulumi_stack_list/
• https://www.pulumi.com/docs/iac/cli/commands/pulumi_stack_remove/

Shell Command Setup

To get started running a shell command on multiple EC2 Instances, head over to AWS Systems Manager via the AWS Console. Then, on the left side under “Node Management”, click on “Run Command”.

On the next screen, click on the orange “Run a Command” button on the right.

Next, under “Command document” search for “shell” to find AWS-RunShellScript and select it.

Then, under “Command parameters” type in the shell command you want to run on your EC2 instances. In my case, I want to run security updates on my instances.

Then select EC2 Instances by your preferred method under “Target Selection”. I chose to manually select my instances but you can also select based on tags or resource groups. Make sure you are using the same region your EC2 instances are in!

This part is optional. If you want log outputs of the commands that are run on every instance you can save logs to an S3 Bucket under “Output options”. Make sure the Instance Profile IAM Role has permissions to write to this bucket!

I left all other settings with their default values.

Now we can click the orange “Run” button on the bottom of the page to execute our command.

After Running a Command

After running a command you’ll see the following screen. You can see the status of the command execution on each EC2 instance to ensure everything went fine. You can also see the Command ID here which will come in handy if you enabled S3 logging.

Viewing Logs

If you enabled logging to S3, you can browse to the bucket you selected and view logs. The output log file will be several directories deep under <Command ID>/<ec2-instance-id>/<command document>/.... It’s easier if you refer to the screenshot below:

I downloaded the stdout file locally and viewed it via the command line to confirm that my command executed properly on this particular EC2 instance:

$ cat stdout

Last metadata expiration check: 0:15:25 ago on Thu Jan 18 21:10:06 2024.
Dependencies resolved.
Nothing to do.
Complete!

Troubleshooting

I personally ran into some issues when trying to do this for the first time. Here are a few troubleshooting solutions.

EC2 Instances Don’t Show Up in Systems Manager

If your EC2 Instances do not show up under “Target selection” they may not have the SSM agent installed. AWS has documentation to guide you through the installation process: Manually installing SSM Agent on EC2 instances for Linux.

If your EC2 Instances do have the SSM agent installed but still aren’t showing up, they may not have an IAM instance profile. You can confirm this by clicking on an EC2 Instance and then checking under the “IAM Role” field. If this field is empty, assign an IAM Role and ensure it has proper permissions. In my case, I used two AWS managed policies AmazonSSMManagedInstanceCore and AmazonSSMPatchAssociation.

(The inline policy in the screenshot is to grant this Role S3 permissions for logging purposes which I cover in the next section.)

No Log Output to S3

If you see no log output in your S3 bucket, you may need to add proper permissions to the instance profile IAM Role of the EC2 instances you selected.

Here is the policy I attached to the IAM instance profile role that got logging to work for me. Change the bucket name my-bucket accordingly!

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetBucketAcl",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ]
    }
  ]
}

find /path/to/starting/directory/ -type d -name "directory_to_delete" -exec rm -rf {} \;

For example, if we wanted to delete all node_modules directories within the path /projects/javascript/, we would run:

find /projects/javascript/ -type d -name "node_modules" -exec rm -rf {} \;

Note:

Sometimes the output says:

find: ./node_modules: No such file or directory

But the command should still work.

If you look closely, we’re really just executing a command against all node_modules directories in the /projects/javascript/ directory (rm -rf). This command can be modified to execute other commands against all node_modules directories, but that is out of the scope of this post :)

With git shortlog you can list all contributor emails for any git repository, not just GitHub repos.

Disclaimer:

I am writing about this to make others aware of this form of scraping and it is purely for educational purposes. I do not plan on doing anything with emails from git repos and you shouldn’t either.

tl;dr:

You can run this command within any git repo to extract all contributor emails:

git shortlog -sea | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" | awk '{print tolower($0)}' | sort | uniq | grep -wv 'users.noreply.github.com'

Command Break Down

The git shortlog -sea part of the command is short for git shortlog --summary --email --all. This command outputs the number of commits each user has made, along with their name and email, across all branches.

$ git shortlog -sea

    54  First Last <FirstLast@example.com>
   385  Another User <Anotheruser@example.com>
     2  user1 <user1@example.com>
    31  first last <firstlast@example.com>
    10  Someone Else <1234567+someoneelse@users.noreply.github.com>

The next command, grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b", extracts emails from each line using a regular expression.

$ git shortlog -sea | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b"

FirstLast@example.com
Anotheruser@example.com
user1@example.com
firstlast@example.com
1234567+someoneelse@users.noreply.github.com

The output from the previous command is piped into awk '{print tolower($0)}', which lowercases all the emails. Sometimes emails are typed in with capital letters. Lowercasing all characters will help with sorting and finding unique emails later.

$ git shortlog -sea | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" | awk '{print tolower($0)}'

firstlast@example.com
anotheruser@example.com
user1@example.com
firstlast@example.com
1234567+someoneelse@users.noreply.github.com

After that, the output is piped into sort and uniq. These commands are straightforward. The emails are sorted alphabetically, then duplicates are excluded from the output.

$ git shortlog -sea | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" | awk '{print tolower($0)}' | sort | uniq

1234567+someoneelse@users.noreply.github.com
anotheruser@example.com
firstlast@example.com
user1@example.com

That should suffice for a lot of git repos, but I also added grep -wv 'users.noreply.github.com' to the end of the command to exclude noreply emails associated with GitHub.

$ git shortlog -sea | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" | awk '{print tolower($0)}' | sort | uniq | grep -wv 'users.noreply.github.com'

anotheruser@example.com
firstlast@example.com
user1@example.com

Extracting Emails With git log

It’s possible to do something similar with the git log --pretty="%ce" command. However, I noticed that this command does not show as many emails as git shortlog. I didn’t look too much into it, but I believe it only pulls emails from one branch rather than all branches like with git shortlog --all.

References

I learned about git shortlog from this Stack Overflow question:

• https://stackoverflow.com/questions/9597410/list-all-developers-on-a-project-in-git

I got the email regex from here:

• https://www.shellhacks.com/regex-find-email-addresses-file-grep/

2024-10-13 Update:

I recently discovered Pearcleaner. You can use this app to delete other apps along with all the extra files and folders they create. I recommend you use this first and then continue reading this post if you want to look more deeply.

You can download Pearcleaner from the link above or install it with Homebrew:

brew install --cask pearcleaner

Searching for Residual Files and Directories

To find directories and files related to LastPass, I ran a system-wide search using find. I excluded directories such as /System/Volumes/Data since those result in errors like “Operation not permitted”. I also excluded Homebrew directories that don’t need to be cleaned up. You can add more directories as needed, just make sure to not add a trailing slash to the filepaths!

find / \
-not \( -path /System/Volumes/Data -prune \) \
-not \( -path /usr/local/Homebrew -prune \) \
-not \( -path /usr/local/Cellar -prune \) \
-not \( -path /usr/local/Caskroom -prune \) \
-name \*lastpass\* 2>&1 | grep -v -E 'Operation not permitted|Permission denied|Not a directory'

The command may take some time to complete depending on your machine specs and amount of files you have. It took around ~10 minutes for me on a 2019 MacBook Pro with an i7 Intel CPU.

Here’s the output I got from the command above. I can now go through each of these files and folders and decide if I want to delete them manually:

/private/var/folders/m9/_7bg6tbn3636m1zzlzq33bwh0000gn/T/com.lastpass.lastpassmacdesktop
/private/var/folders/m9/_7bg6tbn3636m1zzlzq33bwh0000gn/C/com.lastpass.lastpassmacdesktop
/Users/nelson/Library/Application Support/com.lastpass.lastpassmacdesktop
/Users/nelson/Library/WebKit/com.lastpass.lastpassmacdesktop
/Users/nelson/Library/Preferences/com.lastpass.lastpassmacdesktop.plist
/Users/nelson/Library/Application Scripts/N24REP3BMN.lmi.lastpass.group
/Users/nelson/Library/Application Scripts/com.lastpass.lastpassmacdesktop.safariext
/Users/nelson/Library/HTTPStorages/com.lastpass.lastpassmacdesktop.binarycookies
/Users/nelson/Library/HTTPStorages/com.lastpass.lastpassmacdesktop
/Users/nelson/Library/Group Containers/N24REP3BMN.lmi.lastpass.group
/Users/nelson/Library/Group Containers/N24REP3BMN.lmi.lastpass.group/Library/Preferences/N24REP3BMN.lmi.lastpass.group.plist
/Users/nelson/Library/Group Containers/N24REP3BMN.lmi.lastpass.group/Library/Application Scripts/N24REP3BMN.lmi.lastpass.group
/Users/nelson/Library/Containers/com.lastpass.LastPass
/Users/nelson/Library/Containers/com.lastpass.lastpassmacdesktop.safariext
/Users/nelson/Library/Containers/com.lastpass.lastpassmacdesktop.safariext/Data/Library/Application Scripts/com.lastpass.lastpassmacdesktop.safariext
/Users/nelson/Library/Caches/com.crashlytics.data/com.lastpass.lastpassmacdesktop
/Users/nelson/Library/Caches/Homebrew/Cask/lastpass--4.116.0.dmg
/Users/nelson/Library/Caches/com.lastpass.lastpassmacdesktop

This could probably be automated all the way through the deletion step, but I prefer to double check what is actually going to be deleted.

Also, this may help to free up space on macOS, but I mainly did it because I like keeping my system tidy.

Try this out with whatever apps you’ve deleted in the past. You can also try finding files and folders by specifying a company name. For example, replace \*lastpass\* with \*microsoft\* and see what you get!

References

• https://stackoverflow.com/questions/4210042/how-do-i-exclude-a-directory-when-using-find