Building an index

Building is two steps:

1. arcane filter turns a genome FASTA + GTF annotation into a single reference FASTA in which each sequence is labelled with its gene and its exon/intron status.
2. arcane index reads that reference and builds the hash table.

For human and mouse, you can download a prebuilt index. Build your own when you need a different species, a custom annotation, or a different k-mer mask.

Note

The prebuilt indices were built for arcane version 0.1.3 and do not support splice-aware mapping.
If you want to use this option (--mapping-mode splice-aware, see Mapping modes), you currently have to built your own index first.

Step 1: arcane filter

arcane filter \
    --fasta genome.fa.gz \
    --gtf annotation.gtf.gz \
    --mask '####_#_##_###_#_###_###_###_#_###_##_#_####' \
    --name human \
    --prefix index/human

• --fasta is the genome DNA FASTA, --gtf the annotation. To get results most similar to CellRanger, we advise to use the reference and annotation provided by CellRanger!
• --prefix is an output folder (created if needed), not a filename.
• --name is a species/reference label used in the output filenames. A trailing _ is appended if you do not supply one.
• --mask (or -w/--kmersize for a contiguous k-mer) must be given, and must match what you pass to arcane index. Otherwise, the sequence length to get all gapped k-mers than span an exon-junctions is not correct.

The provided example writes into index/human/:

arcane_human_ref.fa.gz               # the reference to feed to `arcane index`
arcane_human_genes.txt               # gene ids, one per line -> --ngenes, matrix rows
arcane_human_new_id_to_original.csv  # gene_number,gene_id,chrom,start,end
arcane_human_mapping_gene_ids.pickle

See File formats for what is inside them.

Selecting a subset of genes

--include-attribute and --exclude-attribute filter on GTF attributes, given as key:value pairs (mutually exclusive — use one or the other):

arcane filter --fasta genome.fa.gz --gtf annotation.gtf.gz \
    --include-attribute gene_type:protein_coding \
    --mask '<mask>' --name human_pc --prefix index/human_pc

--extract_genes

Additionally writes arcane_<NAME>extracted_genes.fa.gz, containing each gene from start to end including introns. Not needed for indexing; useful for inspection.

Step 2: arcane index

arcane index \
    --index myindex \
    --ref index/human/arcane_human_ref.fa.gz \
    --mask '####_#_##_###_#_###_###_###_#_###_##_#_####' \
    -n 2_000_000_000

This writes myindex.hash and myindex.info.

• --ref must be the *ref.fa.gz produced by arcane filter.
• --mask must be identical to the one used in arcane filter.
• -n is required and has no default — see below.

Passing --index

Use the shared prefix (--index myindex), as on this page. Arcane also accepts the .info path (--index myindex.info), which is what the benchmarking workflows in workflow/ do; both work.

Sizing the table with -n

-n/--nobjects is the number of distinct k-mers you expect to store. This depends on your reference and must be estimated beforehand.

If you are unsure, err generously: build with a large -n. As a rough guide, the human genome contains roughly 2.5 billion distinct 25-mers. Since the index has a two step approach, the final index size is independent of the provided n. In the first step, all gapped k-mers are stored in the index. Afterwards, all multi k-mers (those with more colors than supported by the index, not informative for the mapping step) are filtered out and a new (smaller) index with the correct size is built that only contain informative k-mers.

Masks and k-mer size

Arcane indexes gapped k-mers. A mask is a string of # (significant position) and _ (ignored position):

'####_#_##_###_#_###_###_###_#_###_##_#_####'

This one has k=31 significant positions spread over a window of w=43 — the mask used for the prebuilt indices. Quote it in the shell to allow for the underscores. The mask guarantees at least 4 hits (and 51 covered positions) in a read of length~100 with at most 3 substitution differences compared to the reference.

Alternatively, -k/--kmersize INT builds a contiguous k-mer of that length. --mask and -k are mutually exclusive, and exactly one is required.

The mask is stored in the .info file, so arcane map/express use the right one automatically — you do not pass --mask when processing a sample.

--rcmode

• f (default) — forward only. Correct for strand-specific protocols, which is what 10x Genomics scRNA-seq is. For reverse protocols, the reverse complement of the reads is computed during mapping. Use this for single-cell data.
• max — store canonical k-mers, for strand-unspecific protocols (typical bulk RNA-seq).

Note

arcane express only supports indices built for --rcmode f and the direction f and r are supported for mapping. For arcane bulk with strand-specific protocols, we recommend to build the index with --rcmode f and provide the direction (forward, reverse or mix for paired reads, see bulk mode) to the mapping command. For unstranded RNA-seq protocols, we recommend to build a canonical index (--rcmode max) to get more accurate mapping results.

--mature

For sequencing technologies that only capture mature (spliced) mRNA molecules, the --mature option only adds k-mers that appear in transcripts to the index. All intron sequences are ignored. This can slightly increase the mapping rate by preventing multimapping between transcripts and the intronic regions of other genes.

Hash table parameters

Sensible defaults are given in config/index.yaml inside the package and are applied automatically:

bucketsize: 4
fill: 0.95
weakthreads: 8
maxwalk: 500
maxfailures: 0
walkseed: 42
groupprefixlength: 2

You can override any of them on the command line, or point --cfg/--config at your own YAML file (arcane index --cfg my_index_params.yaml --index myindex --ref ... -n 2_000_000_000).

Flag	Meaning
-b, -p, --bucketsize	Elements per bucket. Supplied by the shipped config (4).
--fill	Desired load factor (< 1.0). Slots = ceil(n/fill).
-c, --choices	Number of hash functions (Cuckoo choices). Default 2.
--hashfunctions	default, random, or an explicit colon-separated spec such as linear945:linear9123641:linear349341847. Random functions are chosen per build unless you need reproducibility.
--subtables	Number of subtables; subtables+1 threads are used. Must be odd. Auto-selected if unset.
--statistics	none, summary (default), details, full.
-W, --weakthreads	Threads for computing weak k-mers.
--maxwalk, --maxfailures, --walkseed	Cuckoo insertion behaviour. --maxfailures -1 retries forever.

Gene count for arcane count

If you run the steps separately, arcane count needs --ngenes: the number of lines in the arcane_<NAME>_genes.txt written by arcane filter.

ngenes=$(wc -l < index/human/arcane_human_genes.txt)

arcane express handles this for you.