Htykuut

I am working on developing a single write/reader lock_free queue that is placed on a shared memory which is going to be opened by two different Linux processes. Both processes open the shm with MAP_SHARED.

The queue looks like this:

typedef stuct {

    unsigned char Data[ 256 ];

} Element_Type;

typedef struct {

    unsigned int Snd_Cnt;

    unsigned int Rcv_Cnt;

    Element_Type Elems[ 100 ];

} Queue_Type;

OBS: Both Linux processes open the same shared memory and view it as a pointer to the Queue_Type. Let us say that the pointer is Shm_p.

The Linux process which is the writer does like this:

Tmp_Snd_Cnt = (Shm_p->Snd_Cnt + 1u) % 100;

__atomic_load(&Shm_p>Rcv_Cnt, &Tmp_Rcv_Cnt, __ATOMIC_ACQUIRE);

if (Tmp_Snd_Cnt != Tmp_Rcv_Cnt) {

    memcpy(&Shm_p->Elems[ Tmp_Snd_Cnt ].Data[ 0 ], pointer_to_real_data, size_of_real_data <= 256);

    __atomic_store(&Shm_p>Snd_Cnt, &Tmp_Snd_Cnt, __ATOMIC_RELEASE);

}

The Linux process which is the reader does like this:

Tmp_Rcv_Cnt = Shm_p->Rcv_Cnt;

__atomic_load(&Shm_p>Snd_Cnt, &Tmp_Snd_Cnt, __ATOMIC_ACQUIRE);

if (Tmp_Snd_Cnt != Tmp_Rcv_Cnt) {

    pointer_to_real_data = &Shm_p->Elems[ Tmp_Rcv_Cnt ].Data[ 0 ];

    Tmp_Rcv_Cnt = (Tmp_Rcv_Cnt + 1u) % 100;

    __atomic_store(&Shm_p>Rcv_Cnt, &Tmp_Rcv_Cnt, __ATOMIC_RELEASE);

}

My question is do you see any problems with such an approach like races & stuff?

That what I have seen is the following:
Sometimes the reader gets pointer_to_real_data to point to un-updated data. This suggests that GCC has placed the memcpy after the atomic_store (i.e. compile-time instruction reordering). To mitigate this, I have placed asm volatile("" : : : "memory") right before the atomic store in the writer code. The reader got correct data.

To make things even stranger I took away the asm volatile and compiled again.
The code started to work. (I have cleared the shm before testing again).

So obviously the compiler did something strange in the first case without the asm volatile when the reader got un-updated data. What could have been?

Now, in order to be safe, I have the asm volatile there in order to instruct the compiler to not reorder the code.

Thanks in advance.

LATER EDIT (in order to mitigate the problem with over-writing the pointer_to_real_data in the case when the reader gets slow after the store of Rcv_Cnt):

typedef stuct {

    unsigned char Data[ 256u ];

} Base_Element_Type;

typedef stuct {

    unsigned int      Idx;

    Base_Element_Type Base_Elems[ 2u ];

} Element_Type;

typedef struct {

    unsigned int Snd_Cnt;

    unsigned int Rcv_Cnt;

    Element_Type Elems[ 100u ];

} Queue_Type;

Writer code:

Tmp_Snd_Cnt = (Shm_p->Snd_Cnt + 1u) % 100u;

__atomic_load(&Shm_p>Rcv_Cnt, &Tmp_Rcv_Cnt, __ATOMIC_ACQUIRE);

if (Tmp_Snd_Cnt != Tmp_Rcv_Cnt) {

    Elem_p = &Shm_p->Elems[ Tmp_Snd_Cnt ];

    memcpy(&Elem_p->Base_Elems[ Elem_p->Idx ].Data[ 0u ], pointer_to_real_data, size_of_real_data <= 256u);

    __atomic_store(&Shm_p>Snd_Cnt, &Tmp_Snd_Cnt, __ATOMIC_RELEASE);

}

Reader code:

Tmp_Rcv_Cnt = Shm_p->Rcv_Cnt;

__atomic_load(&Shm_p>Snd_Cnt, &Tmp_Snd_Cnt, __ATOMIC_ACQUIRE);

if (Tmp_Snd_Cnt != Tmp_Rcv_Cnt) {

    Elem_p = &Shm_p->Elems[ Tmp_Rcv_Cnt ];

    pointer_to_real_data = &Elem_p->Base_Elems[ Elem_p->Idx ].Data[ 0u ];

    Elem_p->Idx = (Elem_p->Idx + 1u) % 2u;  

    Tmp_Rcv_Cnt = (Tmp_Rcv_Cnt + 1u) % 100u;

    __atomic_store(&Shm_p>Rcv_Cnt, &Tmp_Rcv_Cnt, __ATOMIC_RELEASE);

}

This should work ... right?!