Shared memory

The index of arcane makes up most of the memory required to process a sample. If you process several samples on one machine at the same time, each process would normally load its own copy of the table, and memory use would scale with the number of samples.

Instead, load the index into POSIX shared memory once and have every process attach to it. The index is then paid for exactly once, no matter how many samples run in parallel.

Load

arcane load --name myindex

--name is the index prefix — the same thing you pass to --index. This maps myindex.hash into a shared memory segment named after the basename of the prefix (here myindex) and reads myindex.info.

Use

Add --shared to arcane express (or arcane map) and pass the same index name:

arcane express --index myindex --shared --R1 $R1 --R2 $R2 --out results/s1 -c v3 &
arcane express --index myindex --shared --R1 $R1 --R2 $R2 --out results/s2 -c v3 &
wait

Remove

The segment persists after arcane load exits, so remember to clean it up:

arcane remove --name myindex

Warning

A loaded index survives until it is explicitly removed or the machine reboots. If you forget, it keeps occupying memory. Check with ls /dev/shm/.

Tying the lifetime to a process

arcane load -k (--keep-running) blocks instead of returning, and removes the segment when it receives SIGTERM or SIGINT. This is convenient in scripts and schedulers, where it guarantees cleanup:

arcane load --name myindex --keep-running &
loader=$!

# ... run samples with --shared ...

kill $loader   # index is removed automatically