Remote Heart Rate Detection using Webcam and 50 Lines of Code
使用網絡攝像頭和 50 行代碼進行遠程心率檢測

Once I came across a description of an Android application that measured the heart rate remotely by using the smartphone’s camera. An interesting point was that the Google Play reviewers did not believe in the possibility of such a measurement, and the application was rejected. I don’t know the end of this story, but it became interesting to check whether this is possible. There is no need to make an Android application, it is much easier to test the idea in Python.
有一次我遇到了一個描述一個 Android 應用程式的故事，它可以通過使用智能手機的攝像頭遠程測量心率。有趣的是，Google Play 的評論者們不相信這種測量的可能性，並且拒絕了這個應用程式。我不知道這個故事的結局，但檢查這是否可能變得有趣。沒有必要製作一個 Android 應用程式，用 Python 測試這個想法要容易得多。

Let’s get started. 讓我們開始吧。

Getting the Camera Stream
獲取相機串流

First, we need to get a stream from the webcam, for which I will use OpenCV. The code is cross-platform and can run on both Windows and Linux/OSX.
首先，我們需要從網絡攝像頭獲取一個流，我將使用 OpenCV。這段代碼是跨平台的，可以在 Windows 和 Linux/OSX 上運行。

import cv2
import io
import time

cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
cap.set(cv2.CAP_PROP_FPS, 30)

while True:
    ret, frame = cap.read()
    img = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the frame
    cv2.imshow('Crop', crop_img)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

The idea of determining the heart rate is that the skin tone changes slightly due to the flow in the blood vessels. So we need a picture crop, which contains only a fragment of the skin:
確定心率的想法是由於血管中的流動，皮膚色調會稍微改變。因此，我們需要一張只包含皮膚片段的圖片裁剪：

x, y, w, h = 800, 500, 100, 100
crop_img = img[y:y + h, x:x + w]

cv2.imshow('Crop', crop_img)

If everything was done correctly, we should get the camera image (blurred for privacy reasons) and crop:
如果一切都做得正確，我們應該能夠獲得攝像頭圖像（出於隱私原因而模糊）並進行裁剪：

Image Processing 圖像處理

Once we have the camera stream, it’s pretty simple. For the selected image fragment, we get the average brightness value and add it to the array along with the measurement timestamp.
一旦我们有了相机流，就很简单了。对于所选的图像片段，我们获取平均亮度值，并将其与测量时间戳一起添加到数组中。

heartbeat_count = 128
heartbeat_values = [0]*heartbeat_count
heartbeat_times = [time.time()]*heartbeat_count

while True:
    ...
    # Update the data and timestamps
    heartbeat_values = heartbeat_values[1:] + [np.average(crop_img)]
    heartbeat_times = heartbeat_times[1:] + [time.time()]

The numpy.average function calculates the average of a two-dimensional array, at the output, we get a number, which is the average brightness of our 100x100 square frame.
numpy.average 函數計算二維數組的平均值，輸出結果是一個數字，該數字表示我們 100x100 正方形框架的平均亮度。

We can display the graph in real-time using the matplotlib library:
我們可以使用matplotlib庫實時顯示圖形：

fig = plt.figure()
ax = fig.add_subplot(111)
while True:
    ...

    ax.plot(heartbeat_times, heartbeat_values)
    fig.canvas.draw()
    plot_img_np = np.fromstring(fig.canvas.tostring_rgb(),
                                dtype=np.uint8, sep='')
    plot_img_np = plot_img_np.reshape(fig.canvas.get_width_height()[::-1] + (3,))
    plt.cla()
    
    cv2.imshow('Graph', plot_img_np)

OpenCV works with images in numpy format, so I have to convert an image using the numpy.fromstring function.
OpenCV 使用 numpy 格式處理圖像，所以我必須使用 numpy.fromstring 函數來轉換圖像。

That’s all. 就這樣。

I have run the program, select such a position so that only a skin fragment is visible in the camera crop area, took The Thinker pose with the head resting on hand — the image should be as motionless as possible. And voila, it really works!
我已經運行了該程序，選擇了這樣一個位置，以便在攝像頭裁剪區域中只有一個皮膚碎片可見，並採取了思考者的姿勢，頭部靠在手上 - 圖像應該盡可能靜止。嗯，真的有效！

It is worth repeating again that the camera is not applied to the skin, we are simply analyzing the overall picture of the person. And it’s amazing that even at this distance, the slight change in skin tone is confidently captured by the camera! As we can see from the graph, the real difference in brightness is less than 0.5% and, of course, it is not visible to the “naked eye”, but confidently distinguishable on the graph. The approximate pulse turned out to be about 75bpm. For comparison, the BPM result from the pulse oximeter:
值得再次重申的是，相機並不應用於皮膚上，我們只是在分析人的整體圖像。令人驚奇的是，即使在這個距離下，相機也能自信地捕捉到皮膚色調的微小變化！從圖表中可以看出，亮度的實際差異小於 0.5％，當然，這對“肉眼”來說是不可見的，但在圖表上可以自信地區分出來。大致脈搏結果約為 75bpm。作為比較，脈搏血氧計的 BPM 結果：

To verify that we get the real heart rate and not any fake signal, like flickering the lightbulb, it is also interesting to check if the heart rate changes depending on the physical exercise. Indeed, I can see it if I put both graphs in the same image — the first graph was taken before the workout, the second after.
為了驗證我們所得到的是真實的心率，而不是任何虛假的信號，例如閃爍的燈泡，檢查心率是否會因身體運動而改變也是很有趣的。事實上，如果我將兩個圖形放在同一張圖像中，我可以看到這一點 - 第一個圖形是在運動前拍攝的，第二個是在運動後拍攝的。

It is easy to see that in the second case the heart rate is higher.
很容易看出，在第二种情况下心率更高。

Conclusion 結論

Oddly enough, but it really works — to be honest, I wasn’t sure about the results. Can it work with a smartphone camera? Well, it depends on many factors. If the phone is placed on the tripod and the person is staying motionless, it can be doable. But if the phone is kept in hand, I strongly doubt that the results will be better than a random number generator ;) On the other side, it may be theoretically possible if the motion stabilization algorithm will be applied to the camera stream before processing, so the question is still open. For the webcam, it is, in general, easier.
奇怪的是，它真的有效-老實說，我對結果不太確定。它能在智能手機相機上運作嗎？嗯，這取決於很多因素。如果手機放在三腳架上，人保持靜止，那麼可能是可行的。但如果手機被拿在手上，我強烈懷疑結果會比隨機數生成器好;) 另一方面，如果在處理之前將運動穩定算法應用於攝像頭流，理論上可能是可行的，所以問題仍然存在。對於網絡攝像頭來說，一般來說更容易。

And since we are analyzing a video stream, a separate question may arise — does this work with compressed video data, is it possible to see the heart rate of a movie or TV actor? I do not know the answer, those who wish can try it on their own. To do this, it is enough to replace the line cap = cv2.VideoCapture(0) in the code with cap = cv2.VideoCapture(“video.mp4”), other code remains the same.
由於我們正在分析視頻流，可能會出現一個單獨的問題 - 是否可以使用壓縮的視頻數據來查看電影或電視演員的心率？我不知道答案，有興趣的人可以自己嘗試。為此，只需將代碼中的cap = cv2.VideoCapture(0)行替換為cap = cv2.VideoCapture(“video.mp4”)，其他代碼保持不變。

For those wishing to make more tests, the source code is attached below:

import numpy as np
from matplotlib import pyplot as plt
import cv2
import io
import time

# Camera stream
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1280)
cap.set(cv2.CAP_PROP_FPS, 30)
# Video stream (optional, not tested)
# cap = cv2.VideoCapture("video.mp4")

# Image crop
x, y, w, h = 800, 500, 100, 100
x, y, w, h = 950, 300, 100, 100
heartbeat_count = 128
heartbeat_values = [0]*heartbeat_count
heartbeat_times = [time.time()]*heartbeat_count

# Matplotlib graph surface
fig = plt.figure()
ax = fig.add_subplot(111)

while True:
    # Capture frame-by-frame
    ret, frame = cap.read()

    img = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    crop_img = img[y:y + h, x:x + w]

    # Update the data
    heartbeat_values = heartbeat_values[1:] + [np.average(crop_img)]
    heartbeat_times = heartbeat_times[1:] + [time.time()]

    # Draw matplotlib graph to numpy array
    ax.plot(heartbeat_times, heartbeat_values)
    fig.canvas.draw()
    plot_img_np = np.fromstring(fig.canvas.tostring_rgb(),
                                dtype=np.uint8, sep='')
    plot_img_np = plot_img_np.reshape(fig.canvas.get_width_height()[::-1] + (3,))
    plt.cla()

    # Display the frames
    cv2.imshow('Crop', crop_img)
    cv2.imshow('Graph', plot_img_np)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Thanks for reading.