16. API - mmalobj

This module provides an object-oriented interface to libmmal which is the library underlying picamera, raspistill, and raspivid. It is provided to ease the usage of libmmal to Python coders unfamiliar with C and also works around some of the idiosyncrasies in libmmal.

Warning

This part of the API is still experimental and subject to change in future versions. Backwards compatibility is not (yet) guaranteed.

16.1. The MMAL Tour

MMAL operates on the principle of pipelines:

  • A pipeline consists of one or more MMAL components (MMALBaseComponent and derivatives) connected together in series.
  • A MMALBaseComponent has one or more ports.
  • A port (MMALControlPort and derivatives) is either a control port, an input port or an output port (there are also clock ports but you generally don’t need to deal with these as MMAL sets them up automatically):
    • Control ports are used to accept and receive commands, configuration parameters, and error messages. All MMAL components have a control port, but in picamera they’re only used for component configuration.
    • Input ports receive data from upstream components.
    • Output ports send data onto downstream components (if they’re connected), or to callback routines in the user’s program (if they’re not connected).
    • Input and output ports can be audio, video or sub-picture (subtitle) ports, but picamera only deals with video ports.
    • Ports have a format which (in the case of video ports) dictates the format of image/frame accepted or generated by the port (YUV, RGB, JPEG, H.264, etc.)
    • Video ports have a framerate which specifies the number of images expected to be received or sent per second.
    • Video ports also have a framesize which specifies the resolution of images/frames accepted or generated by the port.
    • Finally, all ports (control, input and output) have params which affect their operation.
  • An output port can have a MMALConnection to an input port. Connections ensure the two ports use compatible formats, and handle transferring data from output ports to input ports in an orderly fashion. A port cannot have more than one connection from/to it.
  • Data is written to / read from ports via instances of MMALBuffer.
    • Buffers belong to a port and can’t be passed arbitrarily between ports.
    • The size of a buffer is dictated by the format and frame-size of the port that owns the buffer. The memory allocation of a buffer (readable from size) cannot be altered once the port is enabled, but the buffer can contain any amount of data up its allocation size. The actual length of data in a buffer is stored in length.
    • Likewise, the number of buffers belonging to a port is fixed and cannot be altered without disabling the port, reconfiguring it and re-enabling it. The more buffers a port has, the less likely it is that the pipeline will have to drop frames because a component has overrun, but the more GPU memory is required.
    • Buffers also have flags which specify information about the data they contain (e.g. start of frame, end of frame, key frame, etc.)
    • When a connection exists between two ports, the connection continually requests a buffer from the output port, requests another buffer from the input port, copies the output buffer’s data to the input buffer’s data, then returns the buffers to their respective ports (this is a simplification; various tricks are pulled under the covers to minimize the amount of data copying that actually occurs, but as a mental model of what’s going on it’s reasonable).
    • Components take buffers from their input port(s), process them, and write the result into a buffer from the output port(s).

16.1.1. Components

Now we’ve got a mental model of what an MMAL pipeline consists of, let’s build one. For the rest of the tour I strongly recommend using a Pi with a screen (so you can see preview output) but controlling it via an SSH session (so the preview doesn’t cover your command line). Follow along, typing the examples into your remote Python session. And feel free to deviate from the examples if you’re curious about things!

We’ll start by importing the mmalobj module with a convenient alias, then construct a MMALCamera component, and a MMALRenderer component.

>>> from picamera import mmal, mmalobj as mo
>>> camera = mo.MMALCamera()
>>> preview = mo.MMALRenderer()

16.1.2. Ports

Before going any further, let’s have a look at the various ports on these components.

>>> len(camera.inputs)
0
>>> len(camera.outputs)
3
>>> len(preview.inputs)
1
>>> len(preview.outputs)
0

The fact the camera has three outputs should come as little surprise to those who have read the Camera Hardware chapter (if you haven’t already, you might want to skim it now). Let’s examine the first output port of the camera and the input port of the renderer:

>>> camera.outputs[0]
<MMALVideoPort "vc.ril.camera:out:0": format=MMAL_FOURCC('I420')
buffers=1x7680 frames=320x240@0fps>
>>> preview.inputs[0]
<MMALVideoPort "vc.ril.video_render:in:0" format=MMAL_FOURCC('I420')
buffers=2x15360 frames=160x64@0fps>

Several things to note here:

  • We can tell from the port name what sort of component it belongs to, what its index is, and whether it’s an input or an output port
  • Both ports are currently configured for the I420 format; this is MMAL’s name for YUV420 (full resolution Y, quarter resolution UV).
  • The ports have different frame-sizes (320x240 and 160x64 respectively), buffer counts (1 and 2 respectively) and buffer sizes (7680 and 15360 respectively).
  • The buffer sizes look unrealistic. For example, 7680 bytes is nowhere near enough to hold 320 * 240 * 1.5 bytes (YUV420 requires 1.5 bytes per pixel).

Now we’ll configure the camera’s output port with a slightly higher resolution, and give it a frame-rate:

>>> camera.outputs[0].framesize = (640, 480)
>>> camera.outputs[0].framerate = 30
>>> camera.outputs[0].commit()
>>> camera.outputs[0]
<MMALVideoPort "vc.ril.camera:out:0(I420)": format=MMAL_FOURCC('I420')
buffers=1x460800 frames=640x480@30fps>

Note that the changes to the configuration won’t actually take effect until the commit() call. After the port is committed, note that the buffer size now looks reasonable: 640 * 480 * 1.5 = 460800.

16.1.3. Connections

Now we’ll try connecting the renderer’s input to the camera’s output. Don’t worry about the fact that the port configurations are different. One of the nice things about MMAL (and the mmalobj layer) is that connections try very hard to auto-configure things so that they “just work”. Usually, auto-configuration is based upon the output port being connected so it’s important to get that configuration right, but you don’t generally need to worry about the input port.

The renderer is what mmalobj terms a “downstream component”. This is a component with a single input that typically sits downstream from some feeder component (like a camera). All such components have the connect() method which can be used to connect the sole input to a specified output:

>>> preview.connect(camera)
<MMALConnection "vc.ril.camera:out:0/vc.ril.video_render:in:0">
>>> preview.connection.enable()

Note that we’ve been quite lazy in the snippet above by simply calling connect() with the camera component. In this case, a connection will be attempted between the first input port of the owner (preview) and the first unconnected output of the parameter (camera). However, this is not always what’s wanted so you can specify the exact ports you wish to connect. In this case the example was equivalent to calling:

>>> preview.inputs[0].connect(camera.outputs[0])
<MMALConnection "vc.ril.camera:out:0/vc.ril.video_render:in:0">
>>> preview.inputs[0].connection.enable()

Note that the connect() method returns the connection that was constructed but you can also retrieve this by querying the port’s connection attribute later.

As soon as the connection is enabled you should see the camera preview appear on the Pi’s screen. Let’s query the port configurations now:

>>> camera.outputs[0]
<MMALVideoPort "vc.ril.camera:out:0(OPQV)": format=MMAL_FOURCC('OPQV')
buffers=10x128 frames=640x480@30fps>
>>> preview.inputs[0]
<MMALVideoPort "vc.ril.video_render:in:0(OPQV)": format=MMAL_FOURCC('OPQV')
buffers=10x128 frames=640x480@30fps>

Note that the connection has implicitly reconfigured the camera’s output port to use the OPAQUE (“OPQV”) format. This is a special format used internally by the camera firmware which avoids passing complete frame data around, instead passing pointers to frame data around (this explains the tiny buffer size of 128 bytes as very little data is actually being shuttled between the components). Further, note that the connection has automatically copied the port format, frame size and frame-rate to the preview’s input port.

_images/preview_pipeline.svg

16.1.4. Opaque Format

At this point it is worth exploring the differences between the camera’s three output ports:

  • Output 0 is the “preview” output. On this port, the OPAQUE format contains a pointer to a complete frame of data.
  • Output 1 is the “video recording” output. On this port, the OPAQUE format contains a pointer to two complete frames of data. The dual-frame format enables the H.264 video encoder to calculate motion estimation without the encoder having to keep copies of prior frames itself (it can do this when something other than OPAQUE format is used, but dual-image OPAQUE is much more efficient).
  • Output 2 is the “still image” output. On this port, the OPAQUE format contains a pointer to a strip of an image. The “strips” format is used by the JPEG encoder (not to be confused with the MJPEG encoder) to deal with high resolution images efficiently.

Generally, you don’t need to worry about these differences. The mmalobj layer knows about them and negotiates the most efficient format it can for connections. However, they’re worth bearing in mind if you’re aiming to get the most out of the firmware or if you’re confused about why a particular format has been selected for a connection.

16.1.5. Component Configuration

So far we’ve seen how to construct components, configure their ports, and connect them together in rudimentary pipelines. Now, let’s see how to configure components via control port parameters:

>>> camera.control.params[mmal.MMAL_PARAMETER_SYSTEM_TIME]
177572014208
>>> camera.control.params[mmal.MMAL_PARAMETER_SYSTEM_TIME]
177574350658
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS]
Fraction(1, 2)
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS] = 0.75
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS]
Fraction(3, 4)
>>> fx = camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT]
>>> fx
<picamera.mmal.MMAL_PARAMETER_IMAGEFX_T object at 0x765b8440>
>>> dir(fx)
['__class__', '__ctypes_from_outparam__', '__delattr__', '__dict__',
'__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
'__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__',
'__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__',
'__weakref__', '_b_base_', '_b_needsfree_', '_fields_', '_objects', 'hdr',
'value']
>>> fx.value
0
>>> mmal.MMAL_PARAM_IMAGEFX_NONE
0
>>> fx.value = mmal.MMAL_PARAM_IMAGEFX_EMBOSS
>>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = fx
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS] = 1/2
>>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = mmal.MMAL_PARAM_IMAGEFX_NONE
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pi/picamera/picamera/mmalobj.py", line 1109, in __setitem__
    assert mp.hdr.id == key
AttributeError: 'int' object has no attribute 'hdr'
>>> fx.value = mmal.MMAL_PARAM_IMAGEFX_NONE
>>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = fx
>>> preview.disconnect()

Things to note:

  • The parameter dictates the type of the value returned (and accepted, if the parameter is read-write).
  • Many parameters accept a multitude of simple types like int, float, Fraction, str, etc. However, some parameters use ctypes structures and such parameters only accept the relevant structure.
  • The easiest way to use such “structured” parameters is to query them first, modify the resulting structure, then write it back to the parameter.

To find out what parameters are available for use with the camera component, have a look at the source for the PiCamera class, especially property getters and setters.

16.1.6. File Output (RGB capture)

Let’s see how we can produce some file output from the camera. First we’ll perform a straight unencoded RGB capture from the still port (2). As this is unencoded output we don’t need to construct anything else. All we need to do is configure the port for RGB encoding, select an appropriate resolution, then activate the output port:

>>> camera.outputs[2].format = mmal.MMAL_ENCODING_RGB24
>>> camera.outputs[2].framesize = (640, 480)
>>> camera.outputs[2].commit()
>>> camera.outputs[2]
<MMALVideoPort "vc.ril.camera:out:2(RGB3)": format=MMAL_FOURCC('RGB3')
buffers=1x921600 frames=640x480@0fps>
>>> camera.outputs[2].enable()

Unfortunately, that didn’t seem to do much! An output port that is participating in a connection needs nothing more: it knows where its data is going. However, an output port without a connection requires a callback function to be assigned so that something can be done with the buffers of data it produces.

The callback will be given two parameters: the MMALPort responsible for producing the data, and the MMALBuffer containing the data. It is expected to return a bool which will be False if further data is expected and True if no further data is expected. If True is returned, the callback will not be executed again. In our case we’re going to write data out to a file we’ll open before-hand, and we should return True when we see a buffer with the “frame end” flag set:

>>> camera.outputs[2].disable()
>>> import io
>>> output = io.open('image.data', 'wb')
>>> def image_callback(port, buf):
...     output.write(buf.data)
...     return bool(buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END)
...
>>> camera.outputs[2].enable(image_callback)
>>> output.tell()
0

At this stage you may note that while the file exists, nothing’s been written to it. This is because output ports 1 and 2 (the video and still ports) won’t produce any buffers until their “capture” parameter is enabled:

>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
>>> output.tell()
921600
>>> camera.outputs[2].disable()
>>> output.close()

Congratulations! You’ve just captured your first image with the MMAL layer. Given we disconnected the preview above, the current state of the system looks something like this:

_images/rgb_capture_pipeline.svg

16.1.7. File Output (JPEG capture)

Whilst RGB is a useful format for processing we’d generally prefer something like JPEG for output. So, next we’ll construct an MMAL JPEG encoder and use it to compress our RGB capture. Note that we’re not going to connect the JPEG encoder to the camera yet; we’re just going to construct it standalone and feed it data from our capture file, writing the output to another file:

>>> encoder = mo.MMALImageEncoder()
>>> encoder.inputs
(<MMALVideoPort "vc.ril.image_encode:in:0": format=MMAL_FOURCC('RGB2')
buffers=1x15360 frames=96x80@0fps>,)
>>> encoder.outputs
(<MMALVideoPort "vc.ril.image_encode:out:0": format=MMAL_FOURCC('GIF ')
buffers=1x81920 frames=0x0@0fps>,)
>>> encoder.inputs[0].format = mmal.MMAL_ENCODING_RGB24
>>> encoder.inputs[0].framesize = (640, 480)
>>> encoder.inputs[0].commit()
>>> encoder.outputs[0].copy_from(encoder.inputs[0])
>>> encoder.outputs[0]
<MMALVideoPort "vc.ril.image_encode:out:0": format=MMAL_FOURCC('RGB3')
buffers=1x81920 frames=640x480@0fps>
>>> encoder.outputs[0].format = mmal.MMAL_ENCODING_JPEG
>>> encoder.outputs[0].commit()
>>> encoder.outputs[0]
<MMALVideoPort "vc.ril.image_encode:out:0(JPEG)": format=MMAL_FOURCC('JPEG')
buffers=1x307200 frames=0x0@0fps>
>>> encoder.outputs[0].params[mmal.MMAL_PARAMETER_JPEG_Q_FACTOR] = 90

Just pausing for a moment, let’s re-cap what we’ve got: an image encoder constructed, configured for 640x480 RGB input, and JPEG output with a quality factor of “90” (i.e. “very good” - don’t try to read much more than this into JPEG quality settings!). Note that MMAL has set the buffer size at a size it thinks will be typical for the output. As JPEG is a lossy format this won’t be precise and it’s entirely possible that we may receive multiple callbacks for a single frame (if the compression overruns the expected buffer size).

Let’s continue:

>>> rgb_data = io.open('image.data', 'rb')
>>> jpg_data = io.open('image.jpg', 'wb')
>>> def image_callback(port, buf):
...     jpg_data.write(buf.data)
...     return bool(buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END)
...
>>> encoder.outputs[0].enable(image_callback)

16.1.8. File Input (JPEG encoding)

How do we feed data to a component without a connection? We enable its input port with a dummy callback (we don’t need to “do” anything on data input). Then we request buffers from its input port, fill them with data and send them back to the input port:

>>> encoder.inputs[0].enable(lamdba port, buf: True)
>>> buf = encoder.inputs[0].get_buffer()
>>> buf.data = rgb_data.read()
>>> encoder.inputs[0].send_buffer(buf)
>>> jpg_data.tell()
87830
>>> encoder.outputs[0].disable()
>>> encoder.inputs[0].disable()
>>> jpg_data.close()
>>> rgb_data.close()

Congratulations again! You’ve just produced a hardware-accelerated JPEG encoding. The following illustrates the state of the system at the moment (note the camera and renderer still exist; they’re just not connected to anything at the moment):

_images/jpeg_encode_pipeline.svg

Now let’s repeat the process but with the encoder attached to the still port on the camera directly. We can re-use our image_callback routine from earlier and just assign a different output file to jpg_data:

>>> encoder.connect(camera.outputs[2])
<MMALConnection "vc.ril.camera:out:2/vc.ril.image_encode:in:0">
>>> encoder.connection.enable()
>>> encoder.inputs[0]
<MMALVideoPort "vc.ril.image_encode:in:0(OPQV)": format=MMAL_FOURCC('OPQV')
buffers=10x128 frames=640x480@0fps>
>>> jpg_data = io.open('direct.jpg', 'wb')
>>> encoder.outputs[0].enable(image_callback)
>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
>>> jpg_data.tell()
99328
>>> encoder.connection.disable()
>>> jpg_data.close()

Now the state of our system looks like this:

_images/jpeg_capture_pipeline.svg

16.1.9. Threads & Synchronization

The one issue you may have noted is that image_callback is running in a background thread. If we were running our capture extremely fast our main thread might disable the capture before our callback had run. Ideally we want to activate capture, wait on some signal indicating that the callback has completed a single frame successfully, then disable capture. We can do this with the communications primitives from the standard threading module:

>>> from threading import Event
>>> finished = Event()
>>> def image_callback(port, buf):
...     jpg_data.write(buf.data)
...     if buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END:
...         finished.set()
...         return True
...     return False
...
>>> def do_capture(filename='direct.jpg'):
...     global jpg_data
...     jpg_data = io.open(filename, 'wb')
...     finished.clear()
...     encoder.outputs[0].enable(image_callback)
...     camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
...     if not finished.wait(10):
...         raise Exception('capture timed out')
...     camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
...     encoder.outputs[0].disable()
...     jpg_data.close()
...
>>> do_capture()

The above example has several rough edges: globals, no proper clean-up in the case of an exception, etc. but by now you should be getting a pretty good idea of how picamera operates under the hood.

The major difference between picamera and a “typical” MMAL setup is that upon construction, the PiCamera class constructs both a MMALCamera (accessible as PiCamera._camera) and a MMALSplitter (accessible as PiCamera._splitter). The splitter remains permanently attached to the camera’s video port (output port 1). Furthermore, there’s always something connected to the camera’s preview port; by default it’s a MMALNullSink component which is switched with a MMALRenderer when the preview is started.

Encoders are constructed and destroyed as required by calls to capture(), start_recording(), etc. The following illustrates a typical picamera pipeline whilst video recording without a preview:

_images/picamera_pipeline.svg

16.1.10. Debugging Facilities

Before we move onto the pure Python components it’s worth mentioning the debugging capabilities built into mmalobj. Firstly, most objects have useful repr() outputs (in particular, it can be useful to simply evaluate a MMALBuffer to see what flags it’s got and how much data is stored in it). Also, there’s the print_pipeline() function. Give this a port and it’ll dump a human-readable version of your pipeline leading up to that port:

>>> preview.inputs[0].enable(lambda port, buf: True)
>>> buf = preview.inputs[0].get_buffer()
>>> buf
<MMALBuffer object: flags=_____ length=0>
>>> buf.flags = mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END
>>> buf
<MMALBuffer object: flags=E____ length=0>
>>> buf.release()
>>> preview.inputs[0].disable()
>>> mo.print_pipeline(encoder.outputs[0])
 vc.ril.camera [2]                           [0] vc.ril.image_encode [0]
   encoding    OPQV-strips    -->    OPQV-strips      encoding       JPEG
      buf      10x128                     10x128         buf         1x307200
    bitrate    0bps                         0bps       bitrate       0bps
     frame     640x480@0fps         640x480@0fps        frame        0x0@0fps

16.1.11. Python Components

So far all the components we’ve looked at have been “real” MMAL components which is to say that they’re implemented in C, and all talk to bits of the firmware running on the GPU. However, a frequent request has been to be able to modify frames from the camera before they reach the image or video encoder. The Python components are an attempt to make this request relatively simple to achieve from within Python.

The means by which this is achieved are inefficient (to say the least) so don’t expect this to work with high resolutions or framerates. The mmalobj layer in picamera includes the concept of a “Python MMAL” component. To the user these components look a lot like the MMAL components you’ve been playing with above (MMALCamera, MMALImageEncoder, etc). They are instantiated in a similar manner, they have the same sort of ports, and they’re connected using the same means as ordinary MMAL components.

Let’s try this out by placing a transformation between the camera and a preview which will draw a cross over the frames going to the preview. For this we’ll subclass picamera.array.PiArrayTransform. This derives from MMALPythonComponent and provides the useful capability of providing the source and target buffers as numpy arrays containing RGB data:

>>> from picamera import array
>>> class Crosshair(array.PiArrayTransform):
...     def transform(self, source, target):
...         with source as sdata, target as tdata:
...             tdata[...] = sdata
...             tdata[240, :, :] = 0xff
...             tdata[:, 320, :] = 0xff
...         return False
...
>>> transform = Crosshair()

That’s all there is to constructing a transform! This one is a bit crude in as much as the coordinates are hard-coded, and it’s very simplistic, but it should illustrate the principle nicely. Let’s connect it up between the camera and the renderer:

>>> transform.connect(camera)
<MMALPythonConnection "vc.ril.camera.out:0(RGB3)/py.component:in:0">
>>> preview.connect(transform)
<MMALPythonConnection "py.component:out:0/vc.ril.video_render:in:0(RGB3)">
>>> transform.connection.enable()
>>> preview.connection.enable()
>>> transform.enable()

At this point we should take a look at the pipeline to see what’s been configured automatically:

>>> mo.print_pipeline(preview.inputs[0])
 vc.ril.camera [0]                             [0] py.transform [0]                             [0] vc.ril.video_render
   encoding    RGB3            -->            RGB3   encoding   RGB3            -->            RGB3      encoding
      buf      1x921600                   2x921600     buf      2x921600                   2x921600         buf
     frame     640x480@30fps         640x480@30fps    frame     640x480@30fps         640x480@30fps        frame

Apparently the MMAL camera component is outputting RGB data (which is extremely large) to a “py.transform” component, which draws our cross-hair on the buffer and passes it onto the renderer again as RGB. This is part of the inefficiency alluded to above: RGB is a very large format (compared to I420 which is half its size, or OPQV which is tiny) so we’re shuttling a lot of data around here. Expect this to drop frames at higher resolutions or framerates.

The other source of inefficiency isn’t obvious from the debug output above which gives the impression that the “py.transform” component is actually part of the MMAL pipeline. In fact, this is a lie. Under the covers mmalobj installs an output callback on the camera’s output port to feed data to the “py.transform” input port, uses a background thread to run the transform, then copies the results into buffers obtained from the preview’s input port. In other words there’s really two (very short) MMAL pipelines with a hunk of Python running in between them. If mmalobj does its job properly you shouldn’t need to worry about this implementation detail but it’s worth bearing in mind from the perspective of performance.

16.1.12. Performance Hints

Generally you want to your frame handlers to be fast. To avoid dropping frames they’ve got to run in less than a frame’s time (e.g. 33ms at 30fps). Bear in mind that a significant amount of time is going to be spent shuttling the huge RGB frames around so you’ve actually got much less than 33ms available to you (how much will depend on the speed of your Pi, what resolution you’re using, the framerate, etc).

Sometimes, performance can mean making unintuitive choices. For example, the Pillow library (the main imaging library in Python these days) can construct images which share buffer memory (see Image.frombuffer), but only for the indexed (grayscale) and RGBA formats, not RGB. Hence, it can make sense to use RGBA (a format even larger than RGB) if only because it allows you to avoid copying any data when performing a composite.

Another trick is to realize that although YUV420 has different sized planes, it’s often enough to manipulate the Y plane only. In that case you can treat the front of the buffer as an indexed image (remember that Pillow can share buffer memory with such images) and manipulate that directly. With tricks like these it’s possible to perform multiple composites in realtime at 720p30 on a Pi3.

Here’s a (heavily commented) variant of the cross-hair example above that uses the lower level MMALPythonComponent class instead, and the Pillow library to perform compositing on YUV420 in the manner just described:

from picamera import mmal, mmalobj as mo, PiCameraPortDisabled
from PIL import Image, ImageDraw
from signal import pause


class Crosshair(mo.MMALPythonComponent):
    def __init__(self):
        super(Crosshair, self).__init__(name='py.crosshair')
        self._crosshair = None
        self.inputs[0].supported_formats = mmal.MMAL_ENCODING_I420

    def _handle_frame(self, port, buf):
        # If we haven't drawn the crosshair yet, do it now and cache the
        # result so we don't bother doing it again
        if self._crosshair is None:
            self._crosshair = Image.new('L', port.framesize)
            draw = ImageDraw.Draw(self._crosshair)
            draw.line([
                (port.framesize.width // 2, 0),
                (port.framesize.width // 2, port.framesize.height)],
                fill=(255,), width=1)
            draw.line([
                (0, port.framesize.height // 2),
                (port.framesize.width , port.framesize.height // 2)],
                fill=(255,), width=1)
        # buf is the buffer containing the frame from our input port. First
        # we try and grab a buffer from our output port
        try:
            out = self.outputs[0].get_buffer(False)
        except PiCameraPortDisabled:
            # The port was disabled; that probably means we're shutting down so
            # return True to indicate we're all done and the component should
            # be disabled
            return True
        else:
            if out:
                # We've got a buffer (if we don't get a buffer here it most
                # likely means things are going too slow downstream so we'll
                # just have to skip this frame); copy the input buffer to the
                # output buffer
                out.copy_from(buf)
                # now grab a locked reference to the buffer's data by using
                # "with"
                with out as data:
                    # Construct a PIL Image over the Y plane at the front of
                    # the data and tell PIL the buffer is writeable
                    img = Image.frombuffer('L', port.framesize, data, 'raw', 'L', 0, 1)
                    img.readonly = False
                    img.paste(self._crosshair, (0, 0), mask=self._crosshair)
                # Send the output buffer back to the output port so it can
                # continue onward to whatever's downstream
                try:
                    self.outputs[0].send_buffer(out)
                except PiCameraPortDisabled:
                    # The port was disabled; same as before this probably means
                    # we're shutting down so return True to indicate we're done
                    return True
            # Return False to indicate that we want to continue processing
            # frames. If we returned True here, the component would be
            # disabled and no further buffers would be processed
            return False


camera = mo.MMALCamera()
preview = mo.MMALRenderer()
transform = Crosshair()

camera.outputs[0].framesize = '720p'
camera.outputs[0].framerate = 30
camera.outputs[0].commit()

transform.connect(camera)
preview.connect(transform)

transform.connection.enable()
preview.connection.enable()

preview.enable()
transform.enable()
camera.enable()

pause()

It’s a sensible idea to perform any overlay rendering you want to do in a separate thread and then just handle compositing your overlay onto the frame in the MMALPythonComponent._handle_frame() method. Anything you can do to avoid buffer copying is a bonus here.

Here’s a final (rather large) demonstration that puts all these things together to construct a MMALPythonComponent derivative with two purposes:

  1. Render a partially transparent analogue clock in the top left of the frame.
  2. Produces two equivalent I420 outputs; one for feeding to a preview renderer, and another to an encoder (we could use a proper MMAL splitter for this but this is a demonstration of how Python components can have multiple outputs too).
import io
import datetime as dt
from threading import Thread, Lock
from collections import namedtuple
from math import sin, cos, pi
from time import sleep

from picamera import mmal, mmalobj as mo, PiCameraPortDisabled
from PIL import Image, ImageDraw


class Coord(namedtuple('Coord', ('x', 'y'))):
    @classmethod
    def clock_arm(cls, radians):
        return Coord(sin(radians), -cos(radians))

    def __add__(self, other):
        try:
            return Coord(self.x + other[0], self.y + other[1])
        except TypeError:
            return Coord(self.x + other, self.y + other)

    def __sub__(self, other):
        try:
            return Coord(self.x - other[0], self.y - other[1])
        except TypeError:
            return Coord(self.x - other, self.y - other)

    def __mul__(self, other):
        try:
            return Coord(self.x * other[0], self.y * other[1])
        except TypeError:
            return Coord(self.x * other, self.y * other)

    def __floordiv__(self, other):
        try:
            return Coord(self.x // other[0], self.y // other[1])
        except TypeError:
            return Coord(self.x // other, self.y // other)

    # yeah, I could do the rest (truediv, radd, rsub, etc.) but there's no
    # need here...


class ClockSplitter(mo.MMALPythonComponent):
    def __init__(self):
        super(ClockSplitter, self).__init__(name='py.clock', outputs=2)
        self.inputs[0].supported_formats = {mmal.MMAL_ENCODING_I420}
        self._lock = Lock()
        self._clock_image = None
        self._clock_thread = None

    def enable(self):
        super(ClockSplitter, self).enable()
        self._clock_thread = Thread(target=self._clock_run)
        self._clock_thread.daemon = True
        self._clock_thread.start()

    def disable(self):
        super(ClockSplitter, self).disable()
        if self._clock_thread:
            self._clock_thread.join()
            self._clock_thread = None
            with self._lock:
                self._clock_image = None

    def _clock_run(self):
        # draw the clock face up front (no sense drawing that every time)
        origin = Coord(0, 0)
        size = Coord(100, 100)
        center = size // 2
        face = Image.new('L', size)
        draw = ImageDraw.Draw(face)
        draw.ellipse([origin, size - 1], outline=(255,))
        while self.enabled:
            # loop round rendering the clock hands on a copy of the face
            img = face.copy()
            draw = ImageDraw.Draw(img)
            now = dt.datetime.now()
            midnight = now.replace(
                hour=0, minute=0, second=0, microsecond=0)
            timestamp = (now - midnight).total_seconds()
            hour_pos = center + Coord.clock_arm(2 * pi * (timestamp % 43200 / 43200)) * 30
            minute_pos = center + Coord.clock_arm(2 * pi * (timestamp % 3600 / 3600)) * 45
            second_pos = center + Coord.clock_arm(2 * pi * (timestamp % 60 / 60)) * 45
            draw.line([center, hour_pos], fill=(200,), width=2)
            draw.line([center, minute_pos], fill=(200,), width=2)
            draw.line([center, second_pos], fill=(200,), width=1)
            # assign the rendered image to the internal variable
            with self._lock:
                self._clock_image = img
            sleep(0.2)

    def _handle_frame(self, port, buf):
        try:
            out1 = self.outputs[0].get_buffer(False)
            out2 = self.outputs[1].get_buffer(False)
        except PiCameraPortDisabled:
            return True
        if out1:
            # copy the input frame to the first output buffer
            out1.copy_from(buf)
            with out1 as data:
                # construct an Image using the Y plane of the output
                # buffer's data and tell PIL we can write to the buffer
                img = Image.frombuffer('L', port.framesize, data, 'raw', 'L', 0, 1)
                img.readonly = False
                with self._lock:
                    if self._clock_image:
                        img.paste(self._clock_image, (10, 10), self._clock_image)
            # if we've got a second output buffer replicate the first
            # buffer into it (note the difference between replicate and
            # copy_from)
            if out2:
                out2.replicate(out1)
            try:
                self.outputs[0].send_buffer(out1)
            except PiCameraPortDisabled:
                return True
        if out2:
            try:
                self.outputs[1].send_buffer(out2)
            except PiCameraPortDisabled:
                return True
        return False


def main(output_filename):
    camera = mo.MMALCamera()
    preview = mo.MMALRenderer()
    encoder = mo.MMALVideoEncoder()
    clock = ClockSplitter()
    target = mo.MMALPythonTarget(output_filename)

    # Configure camera output 0
    camera.outputs[0].framesize = (640, 480)
    camera.outputs[0].framerate = 24
    camera.outputs[0].commit()

    # Configure H.264 encoder
    encoder.outputs[0].format = mmal.MMAL_ENCODING_H264
    encoder.outputs[0].bitrate = 2000000
    encoder.outputs[0].commit()
    p = encoder.outputs[0].params[mmal.MMAL_PARAMETER_PROFILE]
    p.profile[0].profile = mmal.MMAL_VIDEO_PROFILE_H264_HIGH
    p.profile[0].level = mmal.MMAL_VIDEO_LEVEL_H264_41
    encoder.outputs[0].params[mmal.MMAL_PARAMETER_PROFILE] = p
    encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_INLINE_HEADER] = True
    encoder.outputs[0].params[mmal.MMAL_PARAMETER_INTRAPERIOD] = 30
    encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_INITIAL_QUANT] = 22
    encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_MAX_QUANT] = 22
    encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_MIN_QUANT] = 22

    # Connect everything up and enable everything (no need to enable capture on
    # camera port 0)
    clock.inputs[0].connect(camera.outputs[0])
    preview.inputs[0].connect(clock.outputs[0])
    encoder.inputs[0].connect(clock.outputs[1])
    target.inputs[0].connect(encoder.outputs[0])
    target.connection.enable()
    encoder.connection.enable()
    preview.connection.enable()
    clock.connection.enable()
    target.enable()
    encoder.enable()
    preview.enable()
    clock.enable()
    try:
        sleep(10)
    finally:
        # Disable everything and tear down the pipeline
        target.disable()
        encoder.disable()
        preview.disable()
        clock.disable()
        target.inputs[0].disconnect()
        encoder.inputs[0].disconnect()
        preview.inputs[0].disconnect()
        clock.inputs[0].disconnect()


if __name__ == '__main__':
    main('output.h264')

16.1.13. IO Classes

The Python MMAL components include a couple of useful IO classes: MMALSource and MMALTarget. We could have used these instead of messing around with output callbacks in the sections above but it was worth exploring how those callbacks operated first (in order to comprehend how Python transforms operated).

16.2. Components

class picamera.mmalobj.MMALBaseComponent[source]

Represents a generic MMAL component. Class attributes are read to determine the component type, and the OPAQUE sub-formats of each connectable port.

close()[source]

Close the component and release all its resources. After this is called, most methods will raise exceptions if called.

disable()[source]

Disables the component.

enable()[source]

Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.

control

The MMALControlPort control port of the component which can be used to configure most aspects of the component’s behaviour.

enabled

Returns True if the component is currently enabled. Use enable() and disable() to control the component’s state.

inputs

A sequence of MMALPort objects representing the inputs of the component.

outputs

A sequence of MMALPort objects representing the outputs of the component.

class picamera.mmalobj.MMALCamera[source]

Bases: picamera.mmalobj.MMALBaseComponent

Represents the MMAL camera component. This component has 0 input ports and 3 output ports. The intended use of the output ports (which in turn determines the behaviour of those ports) is as follows:

  • Port 0 is intended for preview renderers
  • Port 1 is intended for video recording
  • Port 2 is intended for still image capture

Use the MMAL_PARAMETER_CAMERA_CONFIG parameter on the control port to obtain and manipulate the camera’s configuration.

annotate_rev

The annotation capabilities of the firmware have evolved over time and several structures are available for querying and setting video annotations. By default the MMALCamera class will pick the latest annotation structure supported by the current firmware but you can select older revisions with annotate_rev for other purposes (e.g. testing).

class picamera.mmalobj.MMALCameraInfo[source]

Bases: picamera.mmalobj.MMALBaseComponent

Represents the MMAL camera-info component. Query the MMAL_PARAMETER_CAMERA_INFO parameter on the control port to obtain information about the connected camera module.

info_rev

The camera information capabilities of the firmware have evolved over time and several structures are available for querying camera information. When initialized, MMALCameraInfo will attempt to discover which structure is in use by the extant firmware. This property can be used to discover the structure version and to modify the version in use for other purposes (e.g. testing).

class picamera.mmalobj.MMALComponent[source]

Bases: picamera.mmalobj.MMALBaseComponent

Represents an MMAL component that acts as a filter of some sort, with a single input that connects to an upstream source port. This is an asbtract base class.

close()[source]

Close the component and release all its resources. After this is called, most methods will raise exceptions if called.

connect(source, **options)[source]

Connects the input port of this component to the specified source MMALPort or MMALPythonPort. Alternatively, as a convenience (primarily intended for command line experimentation; don’t use this in scripts), source can be another component in which case the first unconnected output port will be selected as source.

Keyword arguments will be passed along to the connection constructor. See MMALConnection and MMALPythonConnection for further information.

disable()[source]

Disables the component.

disconnect()[source]

Destroy the connection between this component’s input port and the upstream component.

enable()[source]

Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.

connection

The MMALConnection or MMALPythonConnection object linking this component to the upstream component.

class picamera.mmalobj.MMALSplitter[source]

Bases: picamera.mmalobj.MMALComponent

Represents the MMAL splitter component. This component has 1 input port and 4 output ports which all generate duplicates of buffers passed to the input port.

class picamera.mmalobj.MMALResizer[source]

Bases: picamera.mmalobj.MMALComponent

Represents the MMAL VPU resizer component. This component has 1 input port and 1 output port. This supports resizing via the VPU. This is not as efficient as MMALISPResizer but is available on all firmware verions. The output port can (and usually should) have a different frame size to the input port.

class picamera.mmalobj.MMALISPResizer[source]

Bases: picamera.mmalobj.MMALComponent

Represents the MMAL ISP resizer component. This component has 1 input port and 1 output port, and supports resizing via the VideoCore ISP, along with conversion of numerous formats into numerous other formats (e.g. OPAQUE to RGB, etc). This is more efficient than MMALResizer but is only available on later firmware versions.

class picamera.mmalobj.MMALEncoder[source]

Bases: picamera.mmalobj.MMALComponent

Represents a generic MMAL encoder. This is an abstract base class.

class picamera.mmalobj.MMALVideoEncoder[source]

Bases: picamera.mmalobj.MMALEncoder

Represents the MMAL video encoder component. This component has 1 input port and 1 output port. The output port is usually configured with MMAL_ENCODING_H264 or MMAL_ENCODING_MJPEG.

class picamera.mmalobj.MMALImageEncoder[source]

Bases: picamera.mmalobj.MMALEncoder

Represents the MMAL image encoder component. This component has 1 input port and 1 output port. The output port is typically configured with MMAL_ENCODING_JPEG but can also use MMAL_ENCODING_PNG, MMAL_ENCODING_GIF, etc.

class picamera.mmalobj.MMALDecoder[source]

Bases: picamera.mmalobj.MMALComponent

Represents a generic MMAL decoder. This is an abstract base class.

class picamera.mmalobj.MMALVideoDecoder[source]

Bases: picamera.mmalobj.MMALDecoder

Represents the MMAL video decoder component. This component has 1 input port and 1 output port. The input port is usually configured with MMAL_ENCODING_H264 or MMAL_ENCODING_MJPEG.

class picamera.mmalobj.MMALImageDecoder[source]

Bases: picamera.mmalobj.MMALDecoder

Represents the MMAL iamge decoder component. This component has 1 input port and 1 output port. The input port is usually configured with MMAL_ENCODING_JPEG.

class picamera.mmalobj.MMALRenderer[source]

Bases: picamera.mmalobj.MMALComponent

Represents the MMAL renderer component. This component has 1 input port and 0 output ports. It is used to implement the camera preview and overlays.

class picamera.mmalobj.MMALNullSink[source]

Bases: picamera.mmalobj.MMALComponent

Represents the MMAL null-sink component. This component has 1 input port and 0 output ports. It is used to keep the preview port “alive” (and thus calculating white-balance and exposure) when the camera preview is not required.

16.3. Ports

class picamera.mmalobj.MMALControlPort(port)[source]

Represents an MMAL port with properties to configure the port’s parameters.

disable()[source]

Disable the port.

enable(callback=None)[source]

Enable the port with the specified callback function (this must be None for connected ports, and a callable for disconnected ports).

The callback function must accept two parameters which will be this MMALControlPort (or descendent) and an MMALBuffer instance. Any return value will be ignored.

capabilities

The capabilities of the port. A bitfield of the following:

  • MMAL_PORT_CAPABILITY_PASSTHROUGH
  • MMAL_PORT_CAPABILITY_ALLOCATION
  • MMAL_PORT_CAPABILITY_SUPPORTS_EVENT_FORMAT_CHANGE
enabled

Returns a bool indicating whether the port is currently enabled. Unlike other classes, this is a read-only property. Use enable() and disable() to modify the value.

index

Returns an integer indicating the port’s position within its owning list (inputs, outputs, etc.)

params

The configurable parameters for the port. This is presented as a mutable mapping of parameter numbers to values, implemented by the MMALPortParams class.

type

The type of the port. One of:

  • MMAL_PORT_TYPE_OUTPUT
  • MMAL_PORT_TYPE_INPUT
  • MMAL_PORT_TYPE_CONTROL
  • MMAL_PORT_TYPE_CLOCK
class picamera.mmalobj.MMALPort(port, opaque_subformat='OPQV')[source]

Bases: picamera.mmalobj.MMALControlPort

Represents an MMAL port with properties to configure and update the port’s format. This is the base class of MMALVideoPort, MMALAudioPort, and MMALSubPicturePort.

commit()[source]

Commits the port’s configuration and automatically updates the number and size of associated buffers according to the recommendations of the MMAL library. This is typically called after adjusting the port’s format and/or associated settings (like width and height for video ports).

connect(other, **options)[source]

Connect this port to the other MMALPort (or MMALPythonPort). The type and configuration of the connection will be automatically selected.

Various connection options can be specified as keyword arguments. These will be passed onto the MMALConnection or MMALPythonConnection constructor that is called (see those classes for an explanation of the available options).

copy_from(source)[source]

Copies the port’s format from the source MMALControlPort.

disable()[source]

Disable the port.

disconnect()[source]

Destroy the connection between this port and another port.

enable(callback=None)[source]

Enable the port with the specified callback function (this must be None for connected ports, and a callable for disconnected ports).

The callback function must accept two parameters which will be this MMALControlPort (or descendent) and an MMALBuffer instance. The callback should return True when processing is complete and no further calls are expected (e.g. at frame-end for an image encoder), and False otherwise.

flush()[source]

Flush the port.

get_buffer(block=True, timeout=None)[source]

Returns a MMALBuffer from the associated pool. block and timeout act as they do in the corresponding MMALPool.get_buffer().

send_buffer(buf)[source]

Send MMALBuffer buf to the port.

bitrate

Retrieves or sets the bitrate limit for the port’s format.

buffer_count

The number of buffers allocated (or to be allocated) to the port. The mmalobj layer automatically configures this based on recommendations from the MMAL library.

buffer_size

The size of buffers allocated (or to be allocated) to the port. The size of buffers is typically dictated by the port’s format. The mmalobj layer automatically configures this based on recommendations from the MMAL library.

connection

If this port is connected to another, this property holds the MMALConnection or MMALPythonConnection object which represents that connection. If this port is not connected, this property is None.

format

Retrieves or sets the encoding format of the port. Setting this attribute implicitly sets the encoding variant to a sensible value (I420 in the case of OPAQUE).

After setting this attribute, call commit() to make the changes effective.

opaque_subformat

Retrieves or sets the opaque sub-format that the port speaks. While most formats (I420, RGBA, etc.) mean one thing, the opaque format is special; different ports produce different sorts of data when configured for OPQV format. This property stores a string which uniquely identifies what the associated port means for OPQV format.

If the port does not support opaque format at all, set this property to None.

MMALConnection uses this information when negotiating formats for a connection between two ports.

pool

Returns the MMALPool associated with the buffer, if any.

supported_formats

Retrieves a sequence of supported encodings on this port.

class picamera.mmalobj.MMALVideoPort(port, opaque_subformat='OPQV')[source]

Bases: picamera.mmalobj.MMALPort

Represents an MMAL port used to pass video data.

framerate

Retrieves or sets the framerate of the port’s video frames in fps.

After setting this attribute, call commit() to make the changes effective.

framesize

Retrieves or sets the size of the port’s video frames as a (width, height) tuple. This attribute implicitly handles scaling the given size up to the block size of the camera (32x16).

After setting this attribute, call commit() to make the changes effective.

class picamera.mmalobj.MMALSubPicturePort(port, opaque_subformat='OPQV')[source]

Bases: picamera.mmalobj.MMALPort

Represents an MMAL port used to pass sub-picture (caption) data.

class picamera.mmalobj.MMALAudioPort(port, opaque_subformat='OPQV')[source]

Bases: picamera.mmalobj.MMALPort

Represents an MMAL port used to pass audio data.

class picamera.mmalobj.MMALPortParams(port)[source]

Represents the parameters of an MMAL port. This class implements the MMALControlPort.params attribute.

Internally, the class understands how to convert certain structures to more common Python data-types. For example, parameters that expect an MMAL_RATIONAL_T type will return and accept Python’s Fraction class (or any other numeric types), while parameters that expect an MMAL_BOOL_T type will treat anything as a truthy value. Parameters that expect the MMAL_PARAMETER_STRING_T structure will be treated as plain strings, and likewise MMAL_PARAMETER_INT32_T and similar structures will be treated as plain ints.

Parameters that expect more complex structures will return and expect those structures verbatim.

16.4. Connections

class picamera.mmalobj.MMALBaseConnection(source, target, formats=())[source]

Abstract base class for MMALConnection and MMALPythonConnection. Handles weakrefs to the source and target ports, and format negotiation. All other connection details are handled by the descendent classes.

source

The source MMALPort or MMALPythonPort of the connection.

target

The target MMALPort or MMALPythonPort of the connection.

class picamera.mmalobj.MMALConnection(source, target, formats=default_formats, callback=None)[source]

Bases: picamera.mmalobj.MMALBaseConnection

Represents an MMAL internal connection between two components. The constructor accepts arguments providing the source MMALPort and target MMALPort.

The formats parameter specifies an iterable of formats (in preference order) that the connection may attempt when negotiating formats between the two ports. If this is None, or an empty iterable, no negotiation will take place and the source port’s format will simply be copied to the target port. Otherwise, the iterable will be worked through in order until a format acceptable to both ports is discovered.

Note

The default formats list starts with OPAQUE; the class understands the different OPAQUE sub-formats (see MMAL for more information) and will only select OPAQUE if compatible sub-formats can be used on both ports.

The callback parameter can optionally specify a callable which will be executed for each buffer that traverses the connection (providing an opportunity to manipulate or drop that buffer). If specified, it must be a callable which accepts two parameters: the MMALConnection object sending the data, and the MMALBuffer object containing data. The callable may optionally manipulate the MMALBuffer and return it to permit it to continue traversing the connection, or return None in which case the buffer will be released.

Note

There is a significant performance penalty for specifying a callback between MMAL components as it requires buffers to be copied from the GPU’s memory to the CPU’s memory and back again.

default_formats = (MMAL_ENCODING_OPAQUE, MMAL_ENCODING_I420, MMAL_ENCODING_RGB24, MMAL_ENCODING_BGR24, MMAL_ENCODING_RGBA, MMAL_ENCODING_BGRA)

Class attribute defining the default formats used to negotiate connections between MMAL components.

disable()[source]

Disables the connection.

enable()[source]

Enable the connection. When a connection is enabled, data is continually transferred from the output port of the source to the input port of the target component.

enabled

Returns True if the connection is enabled. Use enable() and disable() to control the state of the connection.

16.5. Buffers

class picamera.mmalobj.MMALBuffer(buf)[source]

Represents an MMAL buffer header. This is usually constructed from the buffer header pointer and is largely supplied to make working with the buffer’s data a bit simpler. Using the buffer as a context manager implicitly locks the buffer’s memory and returns the ctypes buffer object itself:

def callback(port, buf):
    with buf as data:
        # data is a ctypes uint8 array with size entries
        print(len(data))

Alternatively you can use the data property directly, which returns and modifies the buffer’s data as a bytes object (note this is generally slower than using the buffer object unless you are simply replacing the entire buffer):

def callback(port, buf):
    # the buffer contents as a byte-string
    print(buf.data)
acquire()[source]

Acquire a reference to the buffer. This will prevent the buffer from being recycled until release() is called. This method can be called multiple times in which case an equivalent number of calls to release() must be made before the buffer will actually be released.

copy_from(source)[source]

Copies all fields (including data) from the source MMALBuffer. This buffer must have sufficient size to store length bytes from the source buffer. This method implicitly sets offset to zero, and length to the number of bytes copied.

Note

This is fundamentally different to the operation of the replicate() method. It is much slower, but afterward the copied buffer is entirely independent of the source.

copy_meta(source)[source]

Copy meta-data from the source MMALBuffer; specifically this copies all buffer fields with the exception of data, length and offset.

release()[source]

Release a reference to the buffer. This is the opposing call to acquire(). Once all references have been released, the buffer will be recycled.

replicate(source)[source]

Replicates the source MMALBuffer. This copies all fields from the source buffer, including the internal data pointer. In other words, after replication this buffer and the source buffer will share the same block of memory for data.

The source buffer will also be referenced internally by this buffer and will only be recycled once this buffer is released.

Note

This is fundamentally different to the operation of the copy_from() method. It is much faster, but imposes the burden that two buffers now share data (the source cannot be released until the replicant has been released).

reset()[source]

Resets all buffer header fields to default values.

command

The command set in the buffer’s meta-data. This is usually 0 for buffers returned by an encoder; typically this is only used by buffers sent to the callback of a control port.

data

The data held in the buffer as a bytes string. You can set this attribute to modify the data in the buffer. Acceptable values are anything that supports the buffer protocol, and which contains size bytes or less. Setting this attribute implicitly modifies the length attribute to the length of the specified value and sets offset to zero.

Note

Accessing a buffer’s data via this attribute is relatively slow (as it copies the buffer’s data to/from Python objects). See the MMALBuffer documentation for details of a faster (but more complex) method.

dts

The decoding timestamp (DTS) of the buffer, as an integer number of microseconds or MMAL_TIME_UNKNOWN.

flags

The flags set in the buffer’s meta-data, returned as a bitmapped integer. Typical flags include:

  • MMAL_BUFFER_HEADER_FLAG_EOS – end of stream
  • MMAL_BUFFER_HEADER_FLAG_FRAME_START – start of frame data
  • MMAL_BUFFER_HEADER_FLAG_FRAME_END – end of frame data
  • MMAL_BUFFER_HEADER_FLAG_KEYFRAME – frame is a key-frame
  • MMAL_BUFFER_HEADER_FLAG_FRAME – frame data
  • MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO – motion estimatation data
length

The length of data held in the buffer. Must be less than or equal to the allocated size of data held in size minus the data offset. This attribute can be used to effectively blank the buffer by setting it to zero.

offset

The offset from the start of the buffer at which the data actually begins. Defaults to 0. If this is set to a value which would force the current length off the end of the buffer’s size, then length will be decreased automatically.

pts

The presentation timestamp (PTS) of the buffer, as an integer number of microseconds or MMAL_TIME_UNKNOWN.

size

Returns the length of the buffer’s data area in bytes. This will be greater than or equal to length and is fixed in value.

class picamera.mmalobj.MMALQueue(queue)[source]

Represents an MMAL buffer queue. Buffers can be added to the queue with the put() method, and retrieved from the queue (with optional wait timeout) with the get() method.

get(block=True, timeout=None)[source]

Get the next buffer from the queue. If block is True (the default) and timeout is None (the default) then the method will block until a buffer is available. Otherwise timeout is the maximum time to wait (in seconds) for a buffer to become available. If a buffer is not available before the timeout expires, the method returns None.

Likewise, if block is False and no buffer is immediately available then None is returned.

put(buf)[source]

Place MMALBuffer buf at the back of the queue.

put_back(buf)[source]

Place MMALBuffer buf at the front of the queue. This is used when a buffer was removed from the queue but needs to be put back at the front where it was originally taken from.

class picamera.mmalobj.MMALPool(pool)[source]

Represents an MMAL pool containing MMALBuffer objects. All active ports are associated with a pool of buffers, and a queue. Instances can be treated as a sequence of MMALBuffer objects but this is only recommended for debugging purposes; otherwise, use the get_buffer(), send_buffer(), and send_all_buffers() methods which work with the encapsulated MMALQueue.

get_buffer(block=True, timeout=None)[source]

Get the next buffer from the pool’s queue. See MMALQueue.get() for the meaning of the parameters.

resize(new_count, new_size)[source]

Resizes the pool to contain new_count buffers with new_size bytes allocated to each buffer.

new_count must be 1 or more (you cannot resize a pool to contain no headers). However, new_size can be 0 which causes all payload buffers to be released.

Warning

If the pool is associated with a port, the port must be disabled when resizing the pool.

send_all_buffers(port, block=True, timeout=None)[source]

Send all buffers from the queue to port. block and timeout act as they do in get_buffer(). If no buffer is available (for the values of block and timeout, PiCameraMMALError is raised).

send_buffer(port, block=True, timeout=None)[source]

Get a buffer from the pool’s queue and send it to port. block and timeout act as they do in get_buffer(). If no buffer is available (for the values of block and timeout, PiCameraMMALError is raised).

queue

The MMALQueue associated with the pool.

class picamera.mmalobj.MMALPortPool(port)[source]

Bases: picamera.mmalobj.MMALPool

Construct an MMAL pool for the number and size of buffers required by the MMALPort port.

send_all_buffers(port=None, block=True, timeout=None)[source]

Send all buffers from the pool to port (or the port the pool is associated with by default). block and timeout act as they do in MMALPool.get_buffer().

send_buffer(port=None, block=True, timeout=None)[source]

Get a buffer from the pool and send it to port (or the port the pool is associated with by default). block and timeout act as they do in MMALPool.get_buffer().

16.6. Python Extensions

class picamera.mmalobj.MMALPythonPort(owner, port_type, index)[source]

Implements ports for Python-based MMAL components.

commit()[source]

Commits the port’s configuration and automatically updates the number and size of associated buffers. This is typically called after adjusting the port’s format and/or associated settings (like width and height for video ports).

connect(other, **options)[source]

Connect this port to the other MMALPort (or MMALPythonPort). The type and configuration of the connection will be automatically selected.

Various connection options can be specified as keyword arguments. These will be passed onto the MMALConnection or MMALPythonConnection constructor that is called (see those classes for an explanation of the available options).

copy_from(source)[source]

Copies the port’s format from the source MMALControlPort.

disable()[source]

Disable the port.

disconnect()[source]

Destroy the connection between this port and another port.

enable(callback=None)[source]

Enable the port with the specified callback function (this must be None for connected ports, and a callable for disconnected ports).

The callback function must accept two parameters which will be this MMALControlPort (or descendent) and an MMALBuffer instance. Any return value will be ignored.

get_buffer(block=True, timeout=None)[source]

Returns a MMALBuffer from the associated pool. block and timeout act as they do in the corresponding MMALPool.get_buffer().

send_buffer(buf)[source]

Send MMALBuffer buf to the port.

bitrate

Retrieves or sets the bitrate limit for the port’s format.

buffer_count

The number of buffers allocated (or to be allocated) to the port. The default is 2 but more may be required in the case of long pipelines with replicated buffers.

buffer_size

The size of buffers allocated (or to be allocated) to the port. The size of buffers defaults to a value dictated by the port’s format.

capabilities

The capabilities of the port. A bitfield of the following:

  • MMAL_PORT_CAPABILITY_PASSTHROUGH
  • MMAL_PORT_CAPABILITY_ALLOCATION
  • MMAL_PORT_CAPABILITY_SUPPORTS_EVENT_FORMAT_CHANGE
connection

If this port is connected to another, this property holds the MMALConnection or MMALPythonConnection object which represents that connection. If this port is not connected, this property is None.

enabled

Returns a bool indicating whether the port is currently enabled. Unlike other classes, this is a read-only property. Use enable() and disable() to modify the value.

format

Retrieves or sets the encoding format of the port. Setting this attribute implicitly sets the encoding variant to a sensible value (I420 in the case of OPAQUE).

framerate

Retrieves or sets the framerate of the port’s video frames in fps.

framesize

Retrieves or sets the size of the source’s video frames as a (width, height) tuple. This attribute implicitly handles scaling the given size up to the block size of the camera (32x16).

index

Returns an integer indicating the port’s position within its owning list (inputs, outputs, etc.)

pool

Returns the MMALPool associated with the buffer, if any.

supported_formats

Retrieves or sets the set of valid formats for this port. The set must always contain at least one valid format. A single format can be specified; it will be converted implicitly to a singleton set.

If the current port format is not a member of the new set, no error is raised. An error will be raised when commit() is next called if format is still not a member of the set.

type

The type of the port. One of:

  • MMAL_PORT_TYPE_OUTPUT
  • MMAL_PORT_TYPE_INPUT
  • MMAL_PORT_TYPE_CONTROL
  • MMAL_PORT_TYPE_CLOCK
class picamera.mmalobj.MMALPythonBaseComponent[source]

Base class for Python-implemented MMAL components. This class provides the _commit_port() method used by descendents to control their ports’ behaviour, and the enabled property. However, it is unlikely that users will want to sub-class this directly. See MMALPythonComponent for a more useful starting point.

_commit_port(port)[source]

Called by ports when their format is committed. Descendents may override this to reconfigure output ports when input ports are committed, or to raise errors if the new port configuration is unacceptable.

Warning

This method must not reconfigure input ports when called; however it can reconfigure output ports when input ports are committed.

close()[source]

Close the component and release all its resources. After this is called, most methods will raise exceptions if called.

disable()[source]

Disables the component.

enable()[source]

Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.

control

The MMALControlPort control port of the component which can be used to configure most aspects of the component’s behaviour.

enabled

Returns True if the component is currently enabled. Use enable() and disable() to control the component’s state.

inputs

A sequence of MMALPort objects representing the inputs of the component.

outputs

A sequence of MMALPort objects representing the outputs of the component.

class picamera.mmalobj.MMALPythonComponent(name='py.component', outputs=1)[source]

Bases: picamera.mmalobj.MMALPythonBaseComponent

Provides a Python-based MMAL component with a name, a single input and the specified number of outputs (default 1). The connect() and disconnect() methods can be used to establish or break a connection from the input port to an upstream component.

Typically descendents will override the _handle_frame() method to respond to buffers sent to the input port, and will set MMALPythonPort.supported_formats in the constructor to define the formats that the component will work with.

_commit_port(port)[source]

Overridden to to copy the input port’s configuration to the output port(s), and to ensure that the output port(s)’ format(s) match the input port’s format.

_handle_end_of_stream(port, buf)[source]

Handles end-of-stream notifications passed to the component (where MMALBuffer.command is set to MMAL_EVENT_EOS).

The default implementation does nothing but return True (indicating that processing should halt). Override this in descendents to respond to the end of stream.

The port parameter is the port into which the event arrived.

_handle_error(port, buf)[source]

Handles error notifications passed to the component (where MMALBuffer.command is set to MMAL_EVENT_ERROR).

The default implementation does nothing but return True (indicating that processing should halt). Override this in descendents to respond to error events.

The port parameter is the port into which the event arrived.

_handle_format_changed(port, buf)[source]

Handles format change events passed to the component (where MMALBuffer.command is set to MMAL_EVENT_FORMAT_CHANGED).

The default implementation re-configures the input port of the component and emits the event on all output ports for downstream processing. Override this method if you wish to do something else in response to format change events.

The port parameter is the port into which the event arrived, and buf contains the event itself (a MMAL_EVENT_FORMAT_CHANGED_T structure). Use mmal_event_format_changed_get on the buffer’s data to extract the event.

_handle_frame(port, buf)[source]

Handles frame data buffers (where MMALBuffer.command is set to 0).

Typically, if the component has output ports, the method is expected to fetch a buffer from the output port(s), write data into them, and send them back to their respective ports.

Return values are as for normal event handlers (True when no more buffers are expected, False otherwise).

_handle_parameter_changed(port, buf)[source]

Handles parameter change events passed to the component (where MMALBuffer.command is set to MMAL_EVENT_PARAMETER_CHANGED).

The default implementation does nothing but return False (indicating that processing should continue). Override this in descendents to respond to parameter changes.

The port parameter is the port into which the event arrived, and buf contains the event itself (a MMAL_EVENT_PARAMETER_CHANGED_T structure).

close()[source]

Close the component and release all its resources. After this is called, most methods will raise exceptions if called.

connect(source, **options)[source]

Connects the input port of this component to the specified source MMALPort or MMALPythonPort. Alternatively, as a convenience (primarily intended for command line experimentation; don’t use this in scripts), source can be another component in which case the first unconnected output port will be selected as source.

Keyword arguments will be passed along to the connection constructor. See MMALConnection and MMALPythonConnection for further information.

disable()[source]

Disables the component.

disconnect()[source]

Destroy the connection between this component’s input port and the upstream component.

enable()[source]

Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.

connection

The MMALConnection or MMALPythonConnection object linking this component to the upstream component.

class picamera.mmalobj.MMALPythonConnection(source, target, formats=default_formats, callback=None)[source]

Bases: picamera.mmalobj.MMALBaseConnection

Represents a connection between an MMALPythonBaseComponent and a MMALBaseComponent or another MMALPythonBaseComponent. The constructor accepts arguments providing the source MMALPort (or MMALPythonPort) and target MMALPort (or MMALPythonPort).

The formats parameter specifies an iterable of formats (in preference order) that the connection may attempt when negotiating formats between the two ports. If this is None, or an empty iterable, no negotiation will take place and the source port’s format will simply be copied to the target port. Otherwise, the iterable will be worked through in order until a format acceptable to both ports is discovered.

The callback parameter can optionally specify a callable which will be executed for each buffer that traverses the connection (providing an opportunity to manipulate or drop that buffer). If specified, it must be a callable which accepts two parameters: the MMALPythonConnection object sending the data, and the MMALBuffer object containing data. The callable may optionally manipulate the MMALBuffer and return it to permit it to continue traversing the connection, or return None in which case the buffer will be released.

default_formats = (MMAL_ENCODING_I420, MMAL_ENCODING_RGB24, MMAL_ENCODING_BGR24, MMAL_ENCODING_RGBA, MMAL_ENCODING_BGRA)

Class attribute defining the default formats used to negotiate connections between Python and and MMAL components, in preference order. Note that OPAQUE is not present in contrast with the default formats in MMALConnection.

disable()[source]

Disables the connection.

enable()[source]

Enable the connection. When a connection is enabled, data is continually transferred from the output port of the source to the input port of the target component.

enabled

Returns True if the connection is enabled. Use enable() and disable() to control the state of the connection.

class picamera.mmalobj.MMALPythonSource(input)[source]

Bases: picamera.mmalobj.MMALPythonBaseComponent

Provides a source for other MMALComponent instances. The specified input is read in chunks the size of the configured output buffer(s) until the input is exhausted. The wait() method can be used to block until this occurs. If the output buffer is configured to use a full-frame unencoded format (like I420 or RGB), frame-end flags will be automatically generated by the source. When the input is exhausted an empty buffer with the End Of Stream (EOS) flag will be sent.

The component provides all picamera’s usual IO-handling characteristics; if input is a string, a file with that name will be opened as the input and closed implicitly when the component is closed. Otherwise, the input will not be closed implicitly (the component did not open it, so the assumption is that closing input is the caller’s responsibility). If input is an object with a read method it is assumed to be a file-like object and is used as is. Otherwise, input is assumed to be a readable object supporting the buffer protocol (which is wrapped in a BufferIO stream).

close()[source]

Close the component and release all its resources. After this is called, most methods will raise exceptions if called.

disable()[source]

Disables the component.

enable()[source]

Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.

wait(timeout=None)[source]

Wait for the source to send all bytes from the specified input. If timeout is specified, it is the number of seconds to wait for completion. The method returns True if the source completed within the specified timeout and False otherwise.

class picamera.mmalobj.MMALPythonTarget(output, done=1)[source]

Bases: picamera.mmalobj.MMALPythonComponent

Provides a simple component that writes all received buffers to the specified output until a frame with the done flag is seen (defaults to MMAL_BUFFER_HEADER_FLAG_EOS indicating End Of Stream).

The component provides all picamera’s usual IO-handling characteristics; if output is a string, a file with that name will be opened as the output and closed implicitly when the component is closed. Otherwise, the output will not be closed implicitly (the component did not open it, so the assumption is that closing output is the caller’s responsibility). If output is an object with a write method it is assumed to be a file-like object and is used as is. Otherwise, output is assumed to be a writeable object supporting the buffer protocol (which is wrapped in a BufferIO stream).

close()[source]

Close the component and release all its resources. After this is called, most methods will raise exceptions if called.

enable()[source]

Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.

wait(timeout=None)[source]

Wait for the output to be “complete” as defined by the constructor’s done parameter. If timeout is specified it is the number of seconds to wait for completion. The method returns True if the target completed within the specified timeout and False otherwise.

16.7. Debugging

The following functions are useful for quickly dumping the state of a given MMAL pipeline:

picamera.mmalobj.debug_pipeline(port)[source]

Given an MMALVideoPort port, this traces all objects in the pipeline feeding it (including components and connections) and yields each object in turn. Hence the generator typically yields something like:

picamera.mmalobj.print_pipeline(port)[source]

Prints a human readable representation of the pipeline feeding the specified MMALVideoPort port.

Note

It is also worth noting that most classes, in particular MMALVideoPort and MMALBuffer have useful repr() outputs which can be extremely useful with simple print() calls for debugging.

16.8. Utility Functions

The following functions are provided to ease certain common operations in the picamera library. Users of mmalobj may find them handy in various situations:

picamera.mmalobj.open_stream(stream, output=True, buffering=65536)[source]

This is the core of picamera’s IO-semantics. It returns a tuple of a file-like object and a bool indicating whether the stream requires closing once the caller is finished with it.

  • If stream is a string, it is opened as a file object (with mode ‘wb’ if output is True, and the specified amount of bufffering). In this case the function returns (stream, True).
  • If stream is a stream with a write method, it is returned as (stream, False).
  • Otherwise stream is assumed to be a writeable buffer and is wrapped with BufferIO. The function returns (stream, True).
picamera.mmalobj.close_stream(stream, opened)[source]

If opened is True, then the close method of stream will be called. Otherwise, the function will attempt to call the flush method on stream (if one exists). This function essentially takes the output of open_stream() and finalizes the result.

picamera.mmalobj.to_resolution(value)[source]

Converts value which may be a (width, height) tuple or a string containing a representation of a resolution (e.g. “1024x768” or “1080p”) to a (width, height) tuple.

picamera.mmalobj.to_rational(value)[source]

Converts value (which can be anything accepted by to_fraction()) to an MMAL_RATIONAL_T structure.

picamera.mmalobj.buffer_bytes(buf)[source]

Given an object which implements the buffer protocol, this function returns the size of the object in bytes. The object can be multi-dimensional or include items larger than byte-size.