Thilo
This user hasn't shared any biographical information
Homepage: https://thilosdevblog.wordpress.com
How to replace a string in multiple files in linux command line?
Posted in Common on 2022/09/19
Sometimes there are surprises even for long-time developers and linux users. One of these is the commandline tool rpl
: , see https://manpages.ubuntu.com/manpages/jammy/man1/rpl.1.html
You don’t have to struggle with tools like sed
in combination with find
or grep
. Just use a single tool! For example:
rpl -v foo bar '**/*.txt'
It replaces “foo” with “bar” in all .txt files recursively.
How to properly handle the output of the ProcessBuilder
Posted in Coding on 2022/09/18
It’s time for a rewrite of my old post https://thilosdevblog.wordpress.com/2011/11/21/proper-handling-of-the-processbuilder, because since Java 9 there are some new nice features, which make things easier.
There are many topics around the ProcessBuilder. In this post the focus is how to handle the output of executed commands correctly!
The following two things are important:
- Consuming STDOUT and STDERR is necessary to avoid freezing!
- If the process writes a large amount of output, it must be consumed.
This can be done by calling #redirectErrorStream, which redirects STDERR to STDOUT!
Have look at my ProcessBuilderWrapper.
/**
* GPL
*/
public class ProcessBuilderWrapper {
private String[] command;
private File workingDirectory;
// exit value of the process
private int status;
private boolean redirectToStdout;
private boolean redirectErrorStream;
private ByteArrayOutputStream outStd = new ByteArrayOutputStream();
private ByteArrayOutputStream outError = new ByteArrayOutputStream();
// environment variables, that are visible to the process
private Map<String, String> environment = null;
public ProcessBuilderWrapper(String... command) {
this.command = command;
}
public void run() throws IOException, InterruptedException {
ProcessBuilder pb = new ProcessBuilder(command);
if(workingDirectory != null)
pb.directory(workingDirectory);
if(environment != null && environment.size() > 0)
pb.environment().putAll(environment);
pb.redirectErrorStream(redirectErrorStream);
Process process = pb.start();
try (var infoStream = process.getInputStream(); var errorStream = process.getErrorStream()) {
if(redirectToStdout) {
infoStream.transferTo(System.out);
errorStream.transferTo(System.out);
} else {
infoStream.transferTo(this.outStd);
errorStream.transferTo(this.outError);
}
}
status = process.waitFor();
}
public void redirectToStdOut() {
this.redirectToStdout = true;
}
public void redirectErrorStream() {
this.redirectErrorStream = true;
}
public void setWorkingDirectory(String dir) {
this.workingDirectory = Paths.get(dir).toFile();
}
public void setEnvironment(Map<String, String> environment) {
this.environment = environment;
}
public int getStatus() {
return status;
}
public String getOutput() {
return (redirectToStdout) ? "n/a" : outStd.toString();
}
public String getError() {
return (redirectToStdout) ? "n/a" : outError.toString();
}
public boolean hasErrors() {
return getStatus() != 0;
}
}
Usage for regular commands
To run a regular command, you just have to do something like this:
ProcessBuilderWrapper pbw = new ProcessBuilderWrapper("ls", "-al");
pbw.setWorkingDirectory("/tmp");
pbw.redirectErrorStream();
pbw.run();
if(!pbw.hasErrors())
System.out.println(pbw.getOutput());
Time consuming commands
For time consuming commands it could be important to see some progress information or output of the command. In this case, the inputstreams of the Process can be redirected to standard out.
To test my ProcessBuilderWrapper, I used the following time consuming bash script (sleep.sh):
#!/bin/bash
for i in {1..10} ; do
echo "Step: ${i}"
sleep 3s
done
The call of the ProcessBuilderWrapper:
ProcessBuilderWrapper pbw = new ProcessBuilderWrapper("bash", "sleep.sh");
pbw.setWorkingDirectory([path_to_the_script]);
pbw.redirectToStdOut();
pbw.run();
if(!pbw.hasErrors())
System.out.println(pbw.getOutput());
Error case
To test the error case, I used the following bash script (error.sh):
#!/bin/bash
echo "Text on Stdtout"
echo "Text on StdtErr1" >/dev/stderr
echo "Text on StdtErr2" >&2
exit 1
And the call of the ProcessBuilderWrapper:
ProcessBuilderWrapper pbw = new ProcessBuilderWrapper("bash", "error.sh");
pbw.setWorkingDirectory([path_to_the_script]);
pbw.redirectToStdOut();
pbw.run();
if(!pbw.hasErrors())
System.out.println(pbw.getOutput());
It’s tested with mascOS and Linux.
JII (Java Image Info) Version 1.2.0
Posted in Coding on 2022/04/06
Version 1.2.0 of JII has been released.
It provides a clean interface to a collection of Java libraries and source code to read basic properties of images.
Changes:
- updated java to 11
- rewritten the ProcessBuilderWrapper
- fixed call of identify (imagemagick)
- changed repo
Since this version JII gets it’s own maven repository.
The following entries are required in the pom.xml:
<repositories>
<repository>
<id>jiirepo</id>
<url>https://myrepo.thischwa.codes/repository/jii/</url>
</repository>
</repositories>
<dependency>
<groupId>codes.thischwa.jii</groupId>
<artifactId>java-image-info</artifactId>
<version>1.2.0</version>
</dependency>
How to check, if there is an update for mailcow
Let us assume that mailcow (dockerized) is installed in the directory /opt/mailcow-dockerized, then the script below checks, if there is an update and sends an email to the hostmaster:
#!/bin/bash
set -o nounset
set -o errexit
cd /opt/mailcow-dockerized && ./update.sh --check > /dev/null
exit_code=$?
[ $exit_code -eq 0 ] && echo -e "Subject:MAILCOW: Update available \n\n Update your mailcow instance please!\n" | sendmail hostmaster@your-domain.com
exit 0
The scrip can be run as cron.
MySQL: backup databases in separate files and zip it to STDOUT
Posted in Server on 2021/04/03
It’s not very comfortable to restore one database, if you have just one big fat backup file, which contains all databases.
My solution is the bash script above does the following steps:
- It crreates one dump file for each database.
- It packs all dump files in one zipfile
- and sends it to STDOUT.
Step 3 is very helpful e.g. for using this script with backup software like rsnapshot
.
## Dumps every database to a single file, all dump files will be zipped and redirected to the stdout.
## Usage: backup-db.sh > databases.zip
##
#! /bin/bash
set -o nounset
set -o errexit
trap 'rm -rf "$BACKUP_DIR"' EXIT
BACKUP_DIR=$(mktemp -d)
MYSQL=/usr/bin/mysql
MYSQLDUMP=/usr/bin/mysqldump
databases=`$MYSQL --defaults-extra-file=/etc/mysql/credentials.cnf --default-character-set=utf8mb4 -e "SHOW DATABASES;" \
| grep -Ev "(Database|information_schema|performance_schema)"`
for db in $databases; do
$MYSQLDUMP --defaults-extra-file=/etc/mysql/credentials.cnf --default-character-set=utf8mb4 --force --opt --databases $db \
> "$BACKUP_DIR/$db.sql"
done
zip -rjq - "$BACKUP_DIR"
exit 0
The corresponding gist at github: https://gist.github.com/th-schwarz/38dd9fa1cdba05506f642a633095bad7