Shared memory between C and python

Question

I want to share memory between a program in C and another in python.

The c program uses the following structure to define the data.

struct Memory_LaserFrontal {
        char Data[372]; // original data
        float Med[181]; // Measurements in [m]
        charD; // 'I': Invalid -- 'V': Valid
        charS; // 'L': Clean -- 'S': Dirty
        char LaserStatus[2];
        };

From python I have managed to read the variable in memory using sysv_ipc but they have no structure and is seen as a data array. How can I restructure them?

python code:

from time import sleep
import sysv_ipc

# Create shared memory object
memory = sysv_ipc.SharedMemory(1234)

# Read value from shared memory
memory_value = memory.read()

print (memory_value)
print (len(memory_value))
while True:
    memory_value = memory.read()
    print (float(memory_value[800]))
    sleep(0.1)

I have captured and printed the data in python, I have modified the sensor reading and the read data is also modified, confirming that the read data corresponds to the data in the sensor's shared memory. But without the proper structure y cant use the data.

Answer 1

I always like to leave the full source code of the problem solved so others can use it if they have a similar problem. thanks a lot all!

from time import sleep
import sysv_ipc
import struct
import array
# Create shared memory object
while True:
    memory = sysv_ipc.SharedMemory(1234)

    # Read value from shared memory
    memory_value = memory.read()

    #print (memory_value)
    #print (len(memory_value))
    # Get the initial char array - you can turn it into a string if you need to later.
    my_chars = array.array("b") # f for float, c for char etc.

    #my_chars.from_bytes(memory_value[:372]) # happens that 372 chars is 372 bytes.
    Data = my_chars.tolist() # Could be bytes list
    # advance to the member after Data
    end_of_Data = struct.calcsize("372c") 

    # get the length in bytes that 181 floats take up
    end_of_Med = struct.calcsize("181f") + end_of_Data
    # now we know where the floats are
    floats_as_bytes = memory_value[ end_of_Data : end_of_Med ]
    # unpack the remaining parts
    ( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cccc", memory_value[end_of_Med:] )
    print(len(floats_as_bytes)/4)
    a=[]
    for i in range(0,len(floats_as_bytes),4):
        a.append(struct.unpack('<f', floats_as_bytes[i:i+4]))

    print (a[0])
    sleep(0.1)

Answer 2

You need to unpack your binary data structure into Python types. The Python modules struct and array can do this for you.

import struct
import array

NB: Some C compilers, but not the comomn ones, may pad your member variables to align each of them with the expected width for your CPU ( almost always 4 bytes ). This means that it may add padding bytes. You may have to experiment with the struct format parameter 'x' between the appropriate parts of your struct if this is the case. Python's struct module does not expect aligned or padded types by default, you need to inform it. See my note at the very end for a guess on what the padding might look like. Again, per @Max's comment, this is unlikely.

NB: I think the members charD and charS are really char D; and char S;

Assuming you want the floats as a Python list or equivalent we have to do some more work with the Python array module . Same for the char[] Data.

  # Get the initial char array - you can turn it into a string if you need to later.
  my_chars = array.array("b") # f for float, b for byteetc.
  my_chars.from_bytes(memory_value[:372]) # happens that 372 C chars is 372 bytes.
  Data = my_chars.tolist() # Could be bytes list
  # advance to the member after Data
  end_of_Data = struct.calcsize("372c") 
  # get the length in bytes that 181 floats take up
  end_of_Med = struct.calcsize("181f") + end_of_Data
  # now we know where the floats are
  floats_as_bytes = memory_value[ end_of_Data : end_of_Med ]
  # unpack the remaining parts
  ( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cccc", memory_value[end_of_Med:] )

Now use the array module to unpack to make a Python list

   my_floats = array.array("f") # f for float, c for char etc.
   my_floats.from_bytes(floats_as_bytes)

Now Data might be a list of Python bytes type that you need to convert to your preferred string encoding. Usually .decode('utf-8') is good enough.

 Data_S = "".join(Data).decode('utf-8') # get a usable string in Data_S

Padding

struct Memory_LaserFrontal {
        char Data[372]; // 372 is a multiple of 4, probably no padding 
        float Med[181]; // floats are 4 bytes, probably no padding
        charD; // single char, expect 3 padding bytes after
        charS; // single char, expect 3 padding bytes after
        char LaserStatus[2]; // double char expect 2 padding bytes after.
        };

So the last Python line above might be - where the 'x' indicates a padding byte that can be ignored.

  ( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cxxxcxxxccxx", memory_value[end_of_Med:] )