FIFO Shared Memory

Warning

Experimental Feature — The FIFO shared memory mode is experimental and is not enabled by default. All shared memory segments default to a FIFO depth of 1 (standard single-frame behaviour).

When FIFO mode is enabled, the new C, C++, and Python FIFO APIs must be used to read and write data. Using legacy API calls (e.g., accessing image->array.V directly, or calling the old get_data() / get_frame() without understanding FIFO semantics) will produce unexpected behaviour because the underlying memory layout has changed.

Windows is not yet fully tested. The FIFO extension has only been fully tested on Linux and macOS. Windows code which relies on FIFO shared memory should work, but you should be aware that unexpected behaviour is possible.

Overview

The standard DAO shared memory model stores exactly one frame per shared memory segment. The FIFO extension adds a circular buffer of N independent frames (segments) inside a single shared memory object. Each segment has its own copy of IMAGE_METADATA, so timestamps, counters, and the write flag are tracked independently.

A writer advances a shared fifo_last_written index every time a new frame is committed. Each reader keeps its own private read-tail (fifo_last_read / fifo_last_read_cnt0) that lives in the local IMAGE struct and is therefore never visible to other processes. Readers can:

consume frames one-by-one in order (FIFO / sequential mode),
always jump to the newest available frame (latest-only / “newest” mode),
read arbitrary historical segments by index.

Note

Because the read-tail is local to each IMAGE struct, multiple independent readers can each maintain their own position without interfering with each other or with the writer.

Memory Layout

A FIFO SHM object allocates contiguous memory for all N segments in one go:

[ IMAGE_METADATA[0] | IMAGE_METADATA[1] | ... | IMAGE_METADATA[N-1] ]
[ data segment [0]  | data segment [1]  | ... | data segment [N-1]  ]

IMAGE_METADATA[0].fifo_size stores the depth N and is the authoritative value for all processes.
IMAGE_METADATA[0].fifo_last_written is the index of the segment most recently committed by the writer.
Each IMAGE_METADATA[i] holds the counters and flags for segment i.

For a depth-1 FIFO the layout is identical to the legacy format, so existing code that was compiled against the new headers will continue to work as before, as long as it only interacts with depth-1 SHMs. To use SHMs with FIFOs deeper than 1 segment, the code must be updated to use the new FIFO-aware functions.

Return Codes

The FIFO API introduces two new return codes in addition to the existing DAO_SUCCESS, DAO_ERROR, and DAO_TIMEOUT:

Constant	Value	Meaning
`DAO_OVERWRITE`	`-2`	The reader’s tail was lapped by the writer. The tail has been reset to the newest segment. The data pointer returned is still valid but some frames were skipped.
`DAO_NOTREADY`	`-3`	No new segment is available yet (the reader is already at the head of the FIFO).

Creating a FIFO Shared Memory Segment

C API

Use daoShmImageCreate_FIFO instead of daoShmImageCreate:

#include "dao.h"

IMAGE image;
uint32_t size[2] = {256, 256};
uint32_t fifo_depth = 8;   // keep the last 8 frames

// Create a FIFO shared memory segment with depth 8
daoShmImageCreate_FIFO(&image, "/tmp/mydata.im.shm",
                       2,          // naxis
                       size,
                       _DATATYPE_FLOAT,
                       1,          // shared = 1 (POSIX shared memory)
                       0,          // no keywords
                       fifo_depth);

Note

daoShmImageCreate is a thin wrapper around daoShmImageCreate_FIFO with fifo_size = 1 and continues to work as before.

C++ API

Pass the optional depth argument to the Dao::Shm constructor:

#include "daoShm.hpp"

// 256×256 float32 SHM with 8-frame FIFO depth
Dao::Shm<float> shm("/tmp/mydata.im.shm", {256, 256}, nullptr, /*depth=*/8);

Python API

Pass the depth keyword argument when creating a shm object:

import numpy as np
from daoShm import shm

data = np.zeros((256, 256), dtype=np.float32)

# Create SHM with FIFO depth of 8 frames
shm_obj = shm("/tmp/mydata.im.shm", data=data, depth=8)

Writing Data

All existing write functions (daoShmImage2Shm, set_frame(), set_data() etc.) have been updated to advance the FIFO automatically — no API change is required on the writer side. Each call internally increments fifo_last_written and writes into the next circular slot.

Reading Data

C API

Function	Description
`daoShmGetNewestSegment`	Returns the most recently written segment. Equivalent to the old `image->array.V` access pattern.
`daoShmGetNextSegment`	Advances the reader’s tail by one and returns the next unread segment. Returns `DAO_OVERWRITE` if frames were skipped, `DAO_NOTREADY` if no new frame is available.
`daoShmWaitForNextSegment`	Blocks (using the counter semaphore) until a new segment is written, then returns. Call `daoShmGetNextSegment` immediately after.
`daoShmGetArbitrarySegment`	Returns a pointer to a specific FIFO slot by index (wraps modulo `fifo_size`).
`daoShmCheckSegmentOverwrite`	Checks whether the segment last read by this tail has since been overwritten. Returns `DAO_OVERWRITE` if so.
`daoShmResetTail`	Resets the reader’s tail to the newest segment. Useful after an overwrite or on initialisation.

Sequential reading example (C):

#include "dao.h"

IMAGE image;
daoShmShm2Img("/tmp/mydata.im.shm", &image);  // open existing SHM

void   *segment_ptr  = NULL;
uint32_t segment_idx = 0;
uint64_t segment_cnt0 = 0;

while (1) {
    // Block until the next frame arrives
    daoShmWaitForNextSegment(&image);

    int_fast8_t status = daoShmGetNextSegment(&image, &segment_ptr,
                                              &segment_idx, &segment_cnt0);

    if (status == DAO_OVERWRITE) {
        fprintf(stderr, "Warning: frame(s) were overwritten – reader is too slow\n");
    }

    float *data = (float *)segment_ptr;
    // process data …
}

Latest-only reading example (C):

daoShmGetNewestSegment(&image, &segment_ptr, &segment_idx, &segment_cnt0);
float *data = (float *)segment_ptr;

C++ API

Method	Description
`get_frame()`	Returns the newest segment. Existing synchronisation options (semaphore, counter spin) still work.
`get_next_frame(wait, status)`	Advances the tail and returns the next unread segment. `status` is set to `DAO_OVERWRITE` if frames were skipped.
`get_next_frame(wait, status, cnt0)`	As above but also fills `cnt0` with the segment counter value.
`get_arbitrary_frame(segment_idx)`	Returns a pointer to an arbitrary FIFO slot.
`check_segment_overwrite()`	Returns `DAO_OVERWRITE` if the last-read segment has been overwritten.
`get_counter(fifo_idx)`	Returns `cnt0` for a specific FIFO slot.
`get_meta_data(fifo_idx)`	Returns a pointer to `IMAGE_METADATA` for a specific FIFO slot.

Sequential reading example (C++):

#include "daoShm.hpp"

Dao::Shm<float> shm("/tmp/mydata.im.shm");  // open existing SHM

while (true) {
    int_fast8_t status;
    float *data = shm.get_next_frame(/*wait=*/true, status);

    if (status == DAO_OVERWRITE) {
        std::cerr << "Warning: frame(s) overwritten – reader is too slow\n";
    }
    // process data …
}

Python API

Method	Description
`get_data()`	Returns the newest segment as a NumPy array.
`get_data_next(wait, reform)`	Advances the tail and returns the next unread segment. Returns `(status, array)` where status is one of the class constants (`DAO_SUCCESS`, `DAO_OVERWRITE`, `DAO_NOTREADY`).
`get_data_arbitrary(index)`	Returns the segment at the given FIFO index as a NumPy array.
`get_history(num_items)`	Returns the last num_items segments stacked along axis 0 as a single NumPy array.
`get_meta_data(index)`	Returns metadata for the newest segment (`index=None`) or for a specific FIFO slot.

Sequential reading example (Python):

from daoShm import shm

shm_obj = shm("/tmp/mydata.im.shm")  # open existing SHM

while True:
    status, data = shm_obj.get_data_next(wait=True)

    if status == shm.DAO_OVERWRITE:
        print("Warning: frame(s) overwritten – reader is too slow")
    elif status == shm.DAO_NOTREADY:
        continue  # no new frame yet

    # process data (NumPy array) …

Reading recent history (Python):

# Retrieve the last 4 frames as a (4, 256, 256) NumPy array
history = shm_obj.get_history(num_items=4)

Overwrite Detection

Because the FIFO is a circular buffer, a slow reader can be lapped by a fast writer. There are two mechanisms to detect this:

Inline with get_data_next / get_next_frame / daoShmGetNextSegment: The return value is DAO_OVERWRITE whenever the tail was behind the writer by more than one full FIFO cycle. The tail is automatically reset to the newest segment.
Explicit check via check_segment_overwrite / daoShmCheckSegmentOverwrite: Checks whether the write flag or cnt0 of the last-read slot has changed since the read. Useful if you copy data out of shared memory and then want to validate the copy.

To avoid overwrite events, increase the FIFO depth or reduce the time between reader calls.

FIFO Depth Considerations

The primary motivation for using a FIFO depth greater than 1 is retaining a history of frames in shared memory. Two typical use-cases drive the choice of depth:

Telemetry and diagnostics: A telemetry or logging process can read back the last N frames at any time using get_history() / daoShmGetArbitrarySegment. This gives a rolling window of recent data without requiring a separate ring-buffer or file-write on every cycle. The depth should cover at least the longest expected gap between telemetry reads.
Algorithms that depend on previous results: Many adaptive optics algorithms (predictive controllers, integrators, temporal filters) need access to one or more past frames at each iteration. Rather than maintaining separate history buffers inside the component, the FIFO provides this history directly in shared memory, accessible to any process that holds an IMAGE handle. Choose a depth equal to the number of past frames the algorithm requires, plus a margin for scheduling jitter.

The FIFO depth multiplies the shared memory allocation:

Total size ≈ N × (sizeof(IMAGE_METADATA) + nelement × sizeof(element_type))
            + semaphores + keywords

Typical guidance:

Depth 1 — identical to standard (non-FIFO) behaviour. No overhead.
Depth 2–8 — sufficient for algorithms requiring a small number of past frames (e.g., a first- or second-order temporal filter).
Depth 10–100 — suitable for telemetry windows, longer predictive filters, or components that process data in bursts rather than frame-by-frame.
Depth > 100 — feasible for large telemetry histories; bear in mind that the entire buffer lives in RAM. For very long histories consider whether a dedicated file-based ring buffer is more appropriate.

Thread Safety

The writer side (daoShmImage2Shm and equivalents) is safe to call from a single writer thread. Concurrent writers are not supported.
Multiple readers are safe because each reader’s tail is stored in its own local IMAGE struct. Readers never modify shared memory.
The write flag in each segment’s IMAGE_METADATA provides a lightweight consistency indicator but is not a full memory barrier. Use daoShmCheckSegmentOverwrite if strict data integrity is required.

Backwards Compatibility

All existing code that creates SHM with daoShmImageCreate (depth 1) will continue to compile and run correctly — the function now delegates to daoShmImageCreate_FIFO with fifo_size = 1.

Code that loads an existing SHM (daoShmShm2Img) reads fifo_size from the metadata and adjusts all internal offsets automatically, so readers compiled with the new headers can open both depth-1 and depth-N segments transparently.

Warning

SHM segments created with the old library (before the FIFO branch was merged) have fifo_size = 0 in memory, which the new loader interprets incorrectly. Delete and recreate any existing shared memory files after upgrading to the FIFO-enabled build.