The chdb tutorial

The goal of chdb is to execute the same code on a series of input files, and store the results produced by those instances. Using chdb implies that the executions of your code are independent and that the order in which the files are treated is irrelevant. In order to use chdb, you must put all your input files inside a common "input directory". Let’s create 10 files under a directory called inp :

rm -r inp inp.out;
mkdir inp;
for i in $(seq 1 10); do echo "coucou $i" > inp/$i.txt; done

Then let’s use those 10 files as entry for a little toy program. We specify --in-type txt to tell chdb to take all *.txt files as input :

mpirun -n 2 chdb --in-dir inp \
 --in-type txt \
 --command-line 'cat %in-dir%/%path% >%out-dir%/%basename%.out'

A new directory has been created, it is called inp.out and it contains all the output files. If you don’t believe me, just try those commands :

find inp.out -ls
find inp.out -type f |xargs cat

Run again the toy program : it does not work, you get the following message :

ERROR - Cannot create directory inp.out - Error= File exists
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

chdb creates himself the output directory, and refuses to run if the output directory already exists : thus you are protected from destroying by mistake previous computation results

chdb is an mpi application running in master-slave mode. This means that the rank 0 process is the master, it distributes the tasks to the other processes, called "slaves". Those processes are responsible for launching your code. Thus if you request n processes for your chdb job, you will have n-1 slaves running. However, the master should not take too much cpu time, as it only sends some work to the slaves and retrieves their result. So on a 36 cores-node machine, this makes sense to request 37 processes. If you start chdb on 2 nodes, you could request 73 processes, they should be correctly placed on the cores. Please note that the number of slaves running should be less or equal than the number of input files. For instance, the following will not work (2 input files and 3 slaves) :

rm -r inp inp.out;
mkdir inp;
for i in $(seq 1 2); do echo "coucou $i" > inp/$i.txt; done
mpirun -n  4 chdb --in-dir inp --in-type txt \
--command-line 'cat %in-dir%/%path% >%out-dir%/%basename%.out'

try again, replacing -n 4 by -n 3

If you do not provide chdb with any output specification, chdb will build an output directory name for you. But of course it is possible to change this behaviour. This is particularly useful when using chdb through slurm (which is the standard way of working), as you can integrate the job Id in the output directory name :

salloc --time=2:0;
mpirun -n 2 chdb --in-dir inp --in-type txt \
 --command-line 'cat %in-dir%/%path% >%out-dir%/%basename%.out' \
 --out-dir $SLURM_JOB_ID
exit

May be you’ll want to execute the program only on a subset of the files found in the input directory :

Create 10 files in an input directory :

rm -r inp inp.out
mkdir inp
for i in $(seq 1 10); do echo "coucou $i" > inp/$i.txt; done

Create a file containing the list of files to be processed. this will be our subset :

for i in $(seq 1 2 10); do echo $i.txt >> input_files.txt; done

Execute chdb using only the subset as input :

mpirun -n 2 chdb --in-dir inp --in-type txt \
 --command-line 'cat %in-dir%/%path% >%out-dir%/%basename%.out' \
 --in-files input_files.txt;

The h in chdb stands for hierarchy : the input files do not have to be in a "flat directory", they can be organized in a hierarchy, which is sometimes more convenient if you have a lot of files. The following commands will create a hierarchy in the input directory, execute the toy program on the .txt files, then recreate the same hierarchy in output an write here the outputfiles :

rm -r inp inp.out
mkdir -p inp/odd inp/even
for i in $(seq 1 2 9); do echo "coucou $i" > inp/odd/$i.txt;done
for i in $(seq 0 2 8); do echo "coucou $i" > inp/even/$i.txt; done
mpirun -n 2 chdb --in-dir inp --in-type txt \
 --command-line 'cat %in-dir%/%path% >%out-dir%/%dirname%/%basename%.out' \
 --out-files %out-dir%/%dirname%/%basename%.out

Please note the switch --out-files is required here, as it is the only way for chdb to know which subdirectories must be made to recreate the hierarchy in the output directory (you may just try to run the command above without specifying --out-files and see what happens)

You may ask an execution report with the switch --report  :

rm -r inp inp.out;
mkdir inp;
for i in $(seq 1 10); do echo "coucou $i" > inp/$i.txt; done
mpirun -n 3 chdb --in-dir inp --in-type txt \
 --command-line 'sleep 1; cat %in-dir%/%path% >%out-dir%/%basename%.out \
 2>%out-dir%/%basename%.err' \
 --report report.txt

The report gives you some information about :

• The treatment time of every input file
• The computation time of every slave
• Some global data

Those informations are useful to check and improve load balancing, as explained under :

Load balancing is an important feature to be aware of, and may be to improve. You can have some information using the --report switch, as explained above. If the work is well load balanced, all the slaves finish their job at the same time, an no cpu cycles are wasted. In the opposite, if some slaves take more time to complete as others, the faster ones have to wait for the latecomers, and you can loose a lot of cpu time. In the following example, we have have 10 files to treat, and we launch chdb with 4 slaves. each treatment takes about 1s :

rm -r inp inp.out;
mkdir inp;
for i in $(seq 1 10); do echo "coucou $i" > inp/$i.txt; done
mpirun -n 5 chdb --in-dir inp --in-type txt \
 --command-line 'sleep 1; cat %in-dir%/%path% >%out-dir%/%basename%.out \
 2>%out-dir%/%basename%.err' \
 --report report.txt
cat report.txt

Here is the report file :

SLAVE    TIME(s)    STATUS    INPUT PATH
2    1.00965    0    2.txt
1    1.01282    0    1.txt
3    1.01392    0    10.txt
4    1.01529    0    3.txt
2    1.00846    0    4.txt
1    1.00863    0    5.txt
3    1.01006    0    6.txt
4    1.00955    0    7.txt
2    1.0082     0    8.txt
1    1.00727    0    9.txt
----------------------------------------------
SLAVE    N INP    CUMULATED TIME (s)
1    3    3.02872
2    3    3.02631
3    2    2.02398
4    2    2.02484
----------------------------------------------
AVERAGE TIME (s)        = 2.52596
STANDARD DEVIATION (s)  = 0.579144
MIN VALUE (s)           = 2.02398
MAX VALUE (s)           = 3.02872

It is easy to see that 2 slaves work during 2 seconds each, and 2 slaves work during 3 seconds. The first two slaves will have to wait for the last two. But you reserved 4 processors, so you’ll have to "pay" for 4x3=12 seconds, for only 10 useful seconds : 17% cpu time is wasted. In this very simple example, you can easily correct the problem, using 5 slaves instead of 4 :

rm -r inp.out
mpirun -n 6 chdb --in-dir inp --in-type txt --command-line 'sleep 1; cat %in-dir%/%path% >%out-dir%/%basename%.out 2>%out-dir%/%basename%.err' --report report.txt
cat report.txt

The report.txt file shows that this time you will have to "pay" for only 5x2=10 seconds.

SLAVE    TIME(s)    STATUS    INPUT PATH
2    1.01017    0    2.txt
3    1.01334    0    10.txt
1    1.01385    0    1.txt
5    1.01459    0    4.txt
4    1.01562    0    3.txt
2    1.00696    0    5.txt
1    1.0086     0    7.txt
3    1.01       0    6.txt
5    1.01138    0    8.txt
4    1.01139    0    9.txt
----------------------------------------------
SLAVE    N INP    CUMULATED TIME (s)
1    2    2.02245
2    2    2.01713
3    2    2.02334
4    2    2.02701
5    2    2.02597
----------------------------------------------
AVERAGE TIME (s)        = 2.02318
STANDARD DEVIATION (s)  = 0.00386248
MIN VALUE (s)           = 2.01713
MAX VALUE (s)           = 2.02701

This is possible with chdb, you just have to use dir as a file type (consequently dir is a reserved file type and you should not use it for ordinary files). Or course, your code may - or not - be an mpi code.

Here is a sample :

rm -r inpd; for d in $(seq 1 10); do mkdir -p inpd/$d.dir; echo "coucou $d" > inpd/$d.dir/input; done
mpirun -n 11 chdb  --in-dir inpd --in-type dir --command-line 'cat input >output'

The behaviour is somewhat different now :

• There is no default output directory, because we now work in the input directory, which is also the output directory.
• However, you can specify an output directory if this is convenient for you
• While in the previous chdb runs the input directory could have been read only, this is no more the case.

If you program many iterations and generate 1 or more files per iteration, please store them into a bdbh container : you have to call the output directory anything.db and to specify the switch --out-files :

rm -r iter.out.db
mpirun -n 3 chdb  \
 --in-type "1 200" \
 --out-dir iter.out.db  \
 --out-files='%out-dir%/%path%.txt' \
 --command-line 'echo slave nb $CHDB_RANK iteration nb %path% > %out-dir%/%path%.txt'

It is even not required specifying an output directory, as shown here :

rm -r iter.out
mpirun -n 3 chdb \
 --in-type "1 20 2" \
 --command-line 'echo slave nb $CHDB_RANK iteration nb %path%'

The environment variables and the file specification templates are still available in this mode of operation.

WARNING - The feature described here is still considered as EXPERIMENTAL - Feedbacks are welcome If chdb receives a SIGTERM or a SIGKILL signals (for instance if you scancel your running job), the signal is intercepted, and the input files or iterations not yet processed are saved to a file called CHDB-INTERRUPTION.txt. The files being processed when the signal are received are considered as "not processed", because the processing will be interrupted.

It is then easy to launch chdb again, using CHDB-INTERRUPTION.txt as input.

Let’s try it using in-type iteration for more simplicity :

rm -rf iter.out iter1.out
mpirun -n 3 chdb --out-dir iter.out  --in-type "1 100" \
 --command-line 'sleep 1;echo %path% >%out-dir%/%path%'

Execute the above commands, wait about 10 seconds, then type CTRL-C. The program is interrupted, CHDB-INTERRUPTION.txt is created.

First have a look to CHDB-INTERRUPTION.txt, check the last line :

# Number of files not yet processed = 56

If this line is not present, the CHDB-INTERRUPTION.txt is not complete and it can be difficult to resume the execution of your job.

If CHDB-INTERRUPTION.txt is complete, we can launch chdb again using the --in-files switch, and this time we wait until the end :

mpirun -n 3 chdb --out-dir iter1.out  --in-type "1 100" \
 --command-line 'sleep 1;echo %path% >%out-dir%/%path%' \
 --in-files CHDB-INTERRUPTION.txt

You can now check that every file was correctly created ; the following command should display 100 :

find iter.out iter1.out -type f| wc -l