Video Widget Example

Files:

The Video Widget example shows how to implement a video widget using QtMultimedia's QAbstractVideoSurface

VideoWidgetSurface Class Definition

 class VideoWidgetSurface : public QAbstractVideoSurface
 {
     Q_OBJECT
 public:
     VideoWidgetSurface(QWidget *widget, QObject *parent = 0);

     QList<QVideoFrame::PixelFormat> supportedPixelFormats(
             QAbstractVideoBuffer::HandleType handleType = QAbstractVideoBuffer::NoHandle) const;
     bool isFormatSupported(const QVideoSurfaceFormat &format, QVideoSurfaceFormat *similar) const;

     bool start(const QVideoSurfaceFormat &format);
     void stop();

     bool present(const QVideoFrame &frame);

     QRect videoRect() const { return targetRect; }
     void updateVideoRect();

     void paint(QPainter *painter);

 private:
     QWidget *widget;
     QImage::Format imageFormat;
     QRect targetRect;
     QSize imageSize;
     QRect sourceRect;
     QVideoFrame currentFrame;
 };

The VideoWidgetSurface class inherits QAbstractVideoSurface and paints video frames on a QWidget. This is a separate class to VideoWidget as both QAbstractVideoSurface and QWidget inherit QObject.

In addition to the functions from QAbstractVideoSurface, VideoWidgetSurface has functions for determining the video display rectangle, and painting the video.

VideoWidgetSurface Class Implementation

 QList<QVideoFrame::PixelFormat> VideoWidgetSurface::supportedPixelFormats(
         QAbstractVideoBuffer::HandleType handleType) const
 {
     if (handleType == QAbstractVideoBuffer::NoHandle) {
         return QList<QVideoFrame::PixelFormat>()
                 << QVideoFrame::Format_RGB32
                 << QVideoFrame::Format_ARGB32
                 << QVideoFrame::Format_ARGB32_Premultiplied
                 << QVideoFrame::Format_RGB565
                 << QVideoFrame::Format_RGB555;
     } else {
         return QList<QVideoFrame::PixelFormat>();
     }
 }

From the supportedPixelFormats() function we return a list of pixel formats the surface can paint. The order of the list hints at which formats are preferred by the surface. Assuming a 32-bit RGB backbuffer, we'd expect that a 32-bit RGB type with no alpha to be fastest to paint so QVideoFrame::Image_RGB32 is first in the list.

Since we don't support rendering using any special frame handles we don't return any pixel formats if handleType is not QAbstractVideoBuffer::NoHandle.

 bool VideoWidgetSurface::isFormatSupported(
         const QVideoSurfaceFormat &format, QVideoSurfaceFormat *similar) const
 {
     Q_UNUSED(similar);

     const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
     const QSize size = format.frameSize();

     return imageFormat != QImage::Format_Invalid
             && !size.isEmpty()
             && format.handleType() == QAbstractVideoBuffer::NoHandle;
 }

In isFormatSupported() we test if the frame type of a surface format maps to a valid QImage format, that the frame size is not empty, and the handle type is QAbstractVideoBuffer::NoHandle. Note that the QAbstractVideoSurface implementation of isFormatSupported() will verify that the list of supported pixel formats returned by supportedPixelFormats(format.handleType()) contains the pixel format and that the size is not empty so a reimplementation wasn't strictly necessary in this case.

 bool VideoWidgetSurface::start(const QVideoSurfaceFormat &format)
 {
     const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
     const QSize size = format.frameSize();

     if (imageFormat != QImage::Format_Invalid && !size.isEmpty()) {
         this->imageFormat = imageFormat;
         imageSize = size;
         sourceRect = format.viewport();

         QAbstractVideoSurface::start(format);

         widget->updateGeometry();
         updateVideoRect();

         return true;
     } else {
         return false;
     }
 }

To start our surface we'll extract the image format and size from the selected video format and save it for use in the paint() function. If the image format, or size are invalid then we'll set an error and return false. Otherwise we'll save the format and confirm the surface has been started, by calling QAbstractVideoSurface::start(). Finally since the video size may have changed we'll trigger an update of the widget, and video geometry.

 void VideoWidgetSurface::updateVideoRect()
 {
     QSize size = surfaceFormat().sizeHint();
     size.scale(widget->size().boundedTo(size), Qt::KeepAspectRatio);

     targetRect = QRect(QPoint(0, 0), size);
     targetRect.moveCenter(widget->rect().center());
 }

The updateVideoRect() function calculates the region within the widget the video occupies. The size hint of the video format gives a suggested size for the video calculated from the viewport and pixel aspect ratio. If the suggested size fits within the widget then we create a new rect of that size in the center of the widget. Otherwise we shrink the size maintaining the aspect ratio so that it does fit.

 bool VideoWidgetSurface::present(const QVideoFrame &frame)
 {
     if (surfaceFormat().pixelFormat() != frame.pixelFormat()
             || surfaceFormat().frameSize() != frame.size()) {
         setError(IncorrectFormatError);
         stop();

         return false;
     } else {
         currentFrame = frame;

         widget->repaint(targetRect);

         return true;
     }
 }

We can't paint from outside a paint event, so when a new frame is received in present() we save a reference to it and force an immediate repaint of the video region. We retain the saved reference to the frame after the repaint so that the widget can be repainted between frame changes if necessary.

If the format of the frame doesn't match the surface format we can't paint it or very likely any future frames. So we set an UnsupportedFormatError on our surface and stop it immediately.

 void VideoWidgetSurface::paint(QPainter *painter)
 {
     if (currentFrame.map(QAbstractVideoBuffer::ReadOnly)) {
         const QTransform oldTransform = painter->transform();

         if (surfaceFormat().scanLineDirection() == QVideoSurfaceFormat::BottomToTop) {
            painter->scale(1, -1);
            painter->translate(0, -widget->height());
         }

         QImage image(
                 currentFrame.bits(),
                 currentFrame.width(),
                 currentFrame.height(),
                 currentFrame.bytesPerLine(),
                 imageFormat);

         painter->drawImage(targetRect, image, sourceRect);

         painter->setTransform(oldTransform);

         currentFrame.unmap();
     }
 }

The paint() function is called by the video widget to paint the current video frame. Before we draw the frame first we'll check the format for the scan line direction and if the scan lines are arranged from bottom to top we'll flip the painter so the frame isn't drawn upside down. Then using the image format information saved in the start() function we'll construct a new QImage from the current video frame, and draw it to the the widget.

 void VideoWidgetSurface::stop()
 {
     currentFrame = QVideoFrame();
     targetRect = QRect();

     QAbstractVideoSurface::stop();

     widget->update();
 }

When the surface is stopped we need to release the current frame and invalidate the video region. Then we confirm the surface has been stopped by calling QAbstractVideoSurface::stop() which sets the started state to false and finally we update so the video widget so paints over the last frame.

VideoWidget Class Definition

The VideoWidget class uses the VideoWidgetSurface class to implement a video widget.

 class VideoWidget : public QWidget
 {
     Q_OBJECT
 public:
     VideoWidget(QWidget *parent = 0);
     ~VideoWidget();

     QAbstractVideoSurface *videoSurface() const { return surface; }

     QSize sizeHint() const;

 protected:
     void paintEvent(QPaintEvent *event);
     void resizeEvent(QResizeEvent *event);

 private:
     VideoWidgetSurface *surface;
 };

The VideoWidget QWidget implementation is minimal with just the sizeHint(), paintEvent(), and resizeEvent() functions in addition to the constructor, destructor and an instance of VideoWidgetSurface.

VideoWidget Class Implementation

 VideoWidget::VideoWidget(QWidget *parent)
     : QWidget(parent)
     , surface(0)
 {
     setAutoFillBackground(false);
     setAttribute(Qt::WA_NoSystemBackground, true);
     setAttribute(Qt::WA_PaintOnScreen, true);

     QPalette palette = this->palette();
     palette.setColor(QPalette::Background, Qt::black);
     setPalette(palette);

     setSizePolicy(QSizePolicy::MinimumExpanding, QSizePolicy::MinimumExpanding);

     surface = new VideoWidgetSurface(this);
 }

In the VideoWidget constructor we set some flags to speed up re-paints a little. Setting the Qt::WA_NoSystemBackground flag and disabling automatic background fills will stop Qt from a painting a background that'll be completely obscured by the video. The Qt::WA_PaintOnScreen flag will allow us to paint to the screen instead of the back buffer where supported.

Next we set the background color to black, so that any borders around the video are filled in black rather the default background color.

Finally we construct an instance of the VideoWidgetSurface class.

 VideoWidget::~VideoWidget()
 {
     delete surface;
 }

In the destructor we simply delete the VideoWidgetSurface instance.

 QSize VideoWidget::sizeHint() const
 {
     return surface->surfaceFormat().sizeHint();
 }

We get the size hint for the widget from the video format of the surface which is calculated from viewport and pixel aspect ratio of the video format.

 void VideoWidget::paintEvent(QPaintEvent *event)
 {
     QPainter painter(this);

     if (surface->isActive()) {
         const QRect videoRect = surface->videoRect();

         if (!videoRect.contains(event->rect())) {
             QRegion region = event->region();
             region.subtract(videoRect);

             QBrush brush = palette().background();

             foreach (const QRect &rect, region.rects())
                 painter.fillRect(rect, brush);
         }

         surface->paint(&painter);
     } else {
         painter.fillRect(event->rect(), palette().background());
     }
 }

When the video widget receives a paint event we first check if the surface is started, if not then we simply fill the widget with the background color. If it is then we draw a border around the video region clipped by the paint region, before calling paint on the video surface to draw the current frame.

 void VideoWidget::resizeEvent(QResizeEvent *event)
 {
     QWidget::resizeEvent(event);

     surface->updateVideoRect();
 }

The resizeEvent() function is reimplemented to trigger an update of the video region when the widget is resized.

Contents

Video Widget Example

VideoWidgetSurface Class Definition

VideoWidgetSurface Class Implementation

VideoWidget Class Definition

VideoWidget Class Implementation