How to replace a string in multiple files in linux command line?

Sometimes there are surprises even for long-time developers and linux users. One of these is the commandline tool rpl: , see https://manpages.ubuntu.com/manpages/jammy/man1/rpl.1.html

You don’t have to struggle with tools like sed in combination with find or grep. Just use a single tool! For example:

rpl -v foo bar '**/*.txt'

It replaces “foo” with “bar” in all .txt files recursively.

How to properly handle the output of the ProcessBuilder

It’s time for a rewrite of my old post https://thilosdevblog.wordpress.com/2011/11/21/proper-handling-of-the-processbuilder, because since Java 9 there are some new nice features, which make things easier.

There are many topics around the ProcessBuilder. In this post the focus is how to handle the output of executed commands correctly!

The following two things are important:

• Consuming STDOUT and STDERR is necessary to avoid freezing!
• If the process writes a large amount of output, it must be consumed.
  This can be done by calling #redirectErrorStream, which redirects STDERR to STDOUT!

Have look at my ProcessBuilderWrapper.

/**
 * GPL
 */
public class ProcessBuilderWrapper {
	private String[] command;
	private File workingDirectory;
	
	// exit value of the process
	private int status;

	private boolean redirectToStdout;
	private boolean redirectErrorStream;

	private ByteArrayOutputStream outStd = new ByteArrayOutputStream();
	private ByteArrayOutputStream outError = new ByteArrayOutputStream();

	// environment variables, that are visible to the process
	private Map<String, String> environment = null;

	public ProcessBuilderWrapper(String... command) {
		this.command = command;
	}

	public void run() throws IOException, InterruptedException {
		ProcessBuilder pb = new ProcessBuilder(command);
		if(workingDirectory != null)
			pb.directory(workingDirectory);
		if(environment != null && environment.size() > 0)
			pb.environment().putAll(environment);

		pb.redirectErrorStream(redirectErrorStream);

		Process process = pb.start();
		try (var infoStream = process.getInputStream(); var errorStream = process.getErrorStream()) {
			if(redirectToStdout) {
				infoStream.transferTo(System.out);
				errorStream.transferTo(System.out);
			} else {
				infoStream.transferTo(this.outStd);
				errorStream.transferTo(this.outError);
			}
		}
		status = process.waitFor();
	}

	public void redirectToStdOut() {
		this.redirectToStdout = true;
	}

	public void redirectErrorStream() {
		this.redirectErrorStream = true;
	}

	public void setWorkingDirectory(String dir) {
		this.workingDirectory = Paths.get(dir).toFile();
	}

	public void setEnvironment(Map<String, String> environment) {
		this.environment = environment;
	}

	public int getStatus() {
		return status;
	}

	public String getOutput() {
		return (redirectToStdout) ? "n/a" : outStd.toString();
	}

	public String getError() {
		return (redirectToStdout) ? "n/a" : outError.toString();
	}

	public boolean hasErrors() {
		return getStatus() != 0;
	}
}

Usage for regular commands

To run a regular command, you just have to do something like this:

ProcessBuilderWrapper pbw = new ProcessBuilderWrapper("ls", "-al");
pbw.setWorkingDirectory("/tmp");
pbw.redirectErrorStream();
pbw.run();
if(!pbw.hasErrors())
	System.out.println(pbw.getOutput());

Time consuming commands

For time consuming commands it could be important to see some progress information or output of the command. In this case, the inputstreams of the Process can be redirected to standard out.

To test my ProcessBuilderWrapper, I used the following time consuming bash script (sleep.sh):

#!/bin/bash
for i in {1..10} ; do
   echo "Step: ${i}"
   sleep 3s
done

The call of the ProcessBuilderWrapper:

ProcessBuilderWrapper pbw = new ProcessBuilderWrapper("bash", "sleep.sh");
pbw.setWorkingDirectory([path_to_the_script]);
pbw.redirectToStdOut();
pbw.run();
if(!pbw.hasErrors())
	System.out.println(pbw.getOutput());

Error case

To test the error case, I used the following bash script (error.sh):

#!/bin/bash

echo "Text on Stdtout"
echo "Text on StdtErr1" >/dev/stderr
echo "Text on StdtErr2" >&2
exit 1

And the call of the ProcessBuilderWrapper:

ProcessBuilderWrapper pbw = new ProcessBuilderWrapper("bash", "error.sh");
pbw.setWorkingDirectory([path_to_the_script]);
pbw.redirectToStdOut();
pbw.run();
if(!pbw.hasErrors())
	System.out.println(pbw.getOutput());

It’s tested with mascOS and Linux.

JII (Java Image Info) Version 1.2.0

Version 1.2.0 of JII has been released.

It provides a clean interface to a collection of Java libraries and source code to read basic properties of images.

Changes:

• updated java to 11
• rewritten the ProcessBuilderWrapper
• fixed call of identify (imagemagick)
• changed repo

Since this version JII gets it’s own maven repository.

The following entries are required in the pom.xml:

<repositories>
	<repository>
		<id>jiirepo</id>
		<url>https://myrepo.thischwa.codes/repository/jii/</url>
	</repository>
</repositories>

<dependency>
	<groupId>codes.thischwa.jii</groupId>
	<artifactId>java-image-info</artifactId>
	<version>1.2.0</version>
</dependency>

How to check, if there is an update for mailcow

Let us assume that mailcow (dockerized) is installed in the directory /opt/mailcow-dockerized, then the script below checks, if there is an update and sends an email to the hostmaster:

#!/bin/bash
set -o nounset
set -o errexit

cd /opt/mailcow-dockerized && ./update.sh --check > /dev/null
exit_code=$?

[ $exit_code -eq 0 ] && echo -e "Subject:MAILCOW: Update available \n\n Update your mailcow instance please!\n" | sendmail hostmaster@your-domain.com
exit 0

The scrip can be run as cron.

MySQL: backup databases in separate files and zip it to STDOUT

It’s not very comfortable to restore one database, if you have just one big fat backup file, which contains all databases.

My solution is the bash script above does the following steps:

1. It crreates one dump file for each database.
2. It packs all dump files in one zipfile
3. and sends it to STDOUT.

Step 3 is very helpful e.g. for using this script with backup software like rsnapshot.

## Dumps every database to a single file, all dump files will be zipped and redirected to the stdout.
## Usage: backup-db.sh > databases.zip
##
#! /bin/bash
set -o nounset
set -o errexit

trap 'rm -rf "$BACKUP_DIR"' EXIT
BACKUP_DIR=$(mktemp -d)
MYSQL=/usr/bin/mysql
MYSQLDUMP=/usr/bin/mysqldump

databases=`$MYSQL --defaults-extra-file=/etc/mysql/credentials.cnf --default-character-set=utf8mb4 -e "SHOW DATABASES;" \
        | grep -Ev "(Database|information_schema|performance_schema)"`
for db in $databases; do
  $MYSQLDUMP --defaults-extra-file=/etc/mysql/credentials.cnf --default-character-set=utf8mb4 --force --opt --databases $db \
        > "$BACKUP_DIR/$db.sql"
done
zip -rjq - "$BACKUP_DIR"
exit 0

The corresponding gist at github: https://gist.github.com/th-schwarz/38dd9fa1cdba05506f642a633095bad7