Requirement for crowd sensing systems
The need for crowd sensing, especially in urban scenarios, has been acknowledged by the urban governing bodies. The real-time crowd sensing and people counting help urban governing bodies anticipate risks in case of overcrowding. Further, crowd sensing over the long term provides insights into futuristic needs and assists in planning urban infrastructure efficiently. Monitoring the movements of people in narrow lanes and smaller areas can also provide valuable insights. For instance, inputs regarding the areas being used in a community park help provide inputs regarding whether and which play installations in the parks are being used.
Crowd sensing can also help to improve safety in various settings. For example, in bike lanes, overspeeding e-bikes pose a threat to pedestrians and regular bike users [1]. By using crowd sensing to monitor these areas, authorities could help identify and further deter the violators, thus enhancing the road safety.
The need for a reliable crowd sensing system was felt even more during the recent COVID-19 pandemic. In this context, the crowd monitoring systems helped determine whether people were adhering to the curfew timings, and in identifying changing patterns in the movement of people due to pandemic related restrictions [2].
Existing mechanisms for crowd sensing
The popular mechanisms for crowd sensing could broadly be classified into two categories, vision-based crowd sensing and device-based crowd sensing.
Vision-based mechanisms use cameras to capture images and then use algorithms to count the number of people in the image. People-counting cameras are available as hardware products from camera vendors such as HIKvision [3], or as a technology solution from technology vendors such as Canon [4]. Many of these solutions make use of learning algorithms for identifying humans and counting them. There is a vast body of work describing several adaptations of You-Only-Look-Once (YOLO) – a convolutional network based object detection algorithm for people counting [5] [6].
Devise-based solutions estimate the number of people or their movements through their devices. This can include counting Bluetooth devices [7], Wifi access points, or using triangulation of cellular devices to cell phone towers. Google maps, for example, uses location data to understand traffic conditions and suggest routes [8]. As 5G networks become more widespread, celltowers are expected to become even denser, allowing even more precise localization through triangulation.
Challenge with the existing mechanisms for crowd sensing
The advancements in crowd sensing are driven by the need for improvement in accuracy and cost. While there’s a need for high accuracy at a low cost, many of the current approaches – both vision-based and device-based – have the potential for individual identification and tracking. This has led to concerns on privacy intrusion, and consequently, it has led a push in the search for privacy friendly crowd monitoring techniques. It begs a thought whether all the applications indeed require very high accuracy. Is there really a need for high-resolution images just to determine whether a place is over-crowded?
Notably, usage of less intrusive alternatives for data collection is also mandated by GDPR - the European law on data protection on privacy. Article 5(1)(c) specifies that personal data should be “(c) adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed.” (known as ‘data minimisation’) [9]. This makes it important for cities to go for least invasive technology. In fact, the city councils have to meet a higher benchmark, TADA values for instance in case of Amsterdam. This has encouraged the search for alternative crowd sensing technologies that respect the privacy of individuals.
Privacy preserving alternatives
Camera based privacy preserving alternatives
Privacy through camera images could either be addressed at the image capture stage or during the processing stage. The most common approach has been to employ the privacy filtering techniques during the image processing, which allows for the identification of humans and the blurring of their identities before communicating the images upstream. Alternatively, privacy enhancing cameras have been developed that block varying amount of image details based on the privacy level configured [10]. It is possible to configure the cameras to be GDPR compliant, making them communicate only non-personal information such as count of people and their positions.
The second option employs privacy-preserving optics that filter sensitive information and thus balance trade-offs between privacy and the utility of the captured data. Intentional optical defocus has been used for privacy preservation [11]. Thus, the data is anonymized at the capture stage itself, avoiding any privacy concerns at a later stage.
While commercial products employing post-processing for privacy enhancements are available, we have not come across off-the-shelf cameras using optical techniques for blurring personal details.
Solutions employing non-camera techniques
Non-camera alternatives for crowd sensing include the use of radars and WiFi based solutions, as well as LiDARs for mapping urban infrastructure. These provide a high-resolution collection of points, a visual pointcloud, of the scanned area. Commercial LiDAR-based solutions are available for detecting vehicle and crowd movements. For instance, LSR2001BC – a type of LiDAR from the company Conmark - can detect and discriminate between bicycles, pedestrians and scooters [12]. Also, passive Radio- Frequency (RF) can be used to estimate crowd size by inferring information about physical presence of a crowd on communication between RF-devices to infer information regarding the size and/or density of this crowd [13]. Generally, the received signal strength (RSS) is used to estimate the crowd size in such solutions. In addition, Milimeter-wave (mmWave) radar sensors provide a point cloud of the scanned area and can be used to detect people movement and count the passers-by. More detailed information on mmWave is discussed in later sections.
Choice of technology
The choice of the specific technology solution is governed by the application requirements, including the disired accuracy, installation conditions (indoor/outdoor, weather susceptibility), and implementation costs. Legal compliance is also crucial for any urban infrastructure installations. Given the importance of privacy, it is essential to evaluate and consider privacy-friendly alternatives for crowd monitoring. In addition to these technical and legal requirements, the acceptance of the solution as being privacy-friendly is equally important.
The issue of perceived privacy causes many citizens to have strong reservations even with privacy-friendly cameras. Thus, the solutions employing non-camera techniques must be considered for deployment by the city. In this regard, radar-based techniques such as LiDAR and mmWave offer additional details such as distance and velocity information of objects, making them attractive options for certain applications. For example, LiDARs are increasingly used for mapping urban terrain and creating digital elevation models for city planners [14]. They are also being considered for crowd analytics, perimeter security and traffic management [15].
In the following sections of this article, we will explore mmWave for crowd sensing, considering the promise it holds for being a privacy-friendly alternative We also believe that the alternatives listed in Table-1 will continue to co-exist. One of these alternatives will be deployed depending on the application, the extent of accuracy and privacy desired and the budget available for deployment.
Figure 1: Counting performed through image processing algorithms, typically employed by crowd counting cameras
mmWave for crowd sensing
What is mmWave
Millimeter wave (mmWave) is a special class of radar technology that uses wavelengths in the millimeter range, making it a privacy-friendly crowd-sensing alternative. Radar technology is used by mmWave sensors to transmit electromagnetic (EM) wave signals that bounce off objects in their path. Commercial mmWave sensors use a special type of mmWave technology called frequency modulated continuous wave (FMCW) radars, which continuously transmit a frequency-modulated signal, known as a chirp. The comparison of the reflected signal with the transmitted signal is performed in order to measure the range, angle and velocity of the objects [16].
Figure 2: Pointclouds generated by mmWave radar. Privacy friendly point clouds do not reveal any personal information, while providing crowdedness information.
Why mmWave
Being a non-camera based technology, mmWave radar offers better privacy and is more likely to be perceived as a privacy friendly alternative by the general public. The millimeter range wavelength transmitted by the mmWave radars allows for smaller system components, such as antennas, which are required to process the signals. The sparser point cloud resulting from the sparser wavelength of mmWave further guards privacy while still being sufficient for many urban infrastructure applications, such as people counting. Additionally, reports suggest that mmWave is more resilient to weather conditions such as rain and fog as compared to LiDAR and cameras. Notably, the cost of commercially available mmWave sensors are less expensive than LiDAR, making them economically viable for large-scale deployments.
The commercially available mmWave devices are available in 3 different bands, vision-based 24GHz, 60GHz and 77GHz. The high bandwidth band at 24GHz has been phased out for industrial applications due to spectrum regulations and standards [17]. The 77 GHz band is generally reserved for automotive applications. The unlicensed 60 GHz band may be considered for deployment in industrial and infrastructure applications.
Privacy and mmWave
One of the primary privacy design strategies, as suggested in the Little Blue Book, is to limit the processing of personal data, as much as possible. Minimization of the data at collection reduces the possibility of its misuse [18]. The sparse pointcloud created by mmWave adheres to this principle of data minimization.
However, some concerns have been raised about the potential privacy implications of mmWave technology. There have been reports suggesting the usage of mmWave radars for measuring personal parameters such as monitoring heartbeats and performing gait analysis. This creates a perception that mmWave too may be privacy-invasive.
The level of privacy provided by a technology has to be interpreted together with the way the device is constructed and configured. For example, the resolution provided by mmWave depends upon the number of transmitter and receiver antenna deployed for scanning, their physical pattern and the configuration parameters. Thus the resolution could be controlled by construction. Further, the configuration is typically a trade-off between the achievable range and the resolution. The works on Gait-analysis based person identification have quoted an accuracy of around 90%, but for a limited set of 10-12 people and under indoor controlled conditions [19], [20]. Similarly, in case of critical parameter measurements, the subject is expected to be stationary, with a certain posture and within a few metres of the radar [21]. These scenarios are very different from deployment in urban infrastructure, where the scene is dynamic and posture of people is not controlled. Further, numerous people may walk through a street in one single day, thus making identification difficult through learning techniques.
“MmWave sensors have potential to become the privacy-friendly alternative to camera surveillance in order to count the number of people in crowded areas without identifying anyone.”
Rynaldo Koerhuis, Privacy Officer, Municipality of Amsterdam
Is mmWave safe
The uncanny similarity of the term mmWave with microwave raises certain inhibitions against its usage. The kitchen oven works with microwave technology and heats and cooks food. Hence, will the use of mmWave cause any health hazards?
Firstly, mmWave frequencies are considered non-ionizing radiation, which means they are not cancerous. Secondly, while kitchen microwaves use high energy levels to heat food in a small, enclosed space, commercially available mmWave devices emit much lower power in the milliwatt range. These devices are also required to comply with strict power density regulations set by organizations like the FCC and ICNRP, ensuring their safety for public use.
Finally, independent research into the potential health hazards of mmWave has found no adverse effects at the permitted power levels. Thus, it could be concluded that the commercial mmWave devices are safe to use in public infrastructure.
Pilot installation for mmWave technology
The benefits of mmWave technology make it an ideal candidate for urban infrastructure deployment. Commercial devices are readily available, making it easier to prototype and potentially reduce costs. This presents a compelling case for deploying mmWave sensors for crowd sensing, which requires investigating several factors, which will be elaborated upon below.
Suitable enclosure for mmWave
The mmWave sensor must be enclosed in a suitable cover to protect it from the environment when deployed outdoors. The enclosure material and shape, known as a radome, must preserve the characteristics of the mmWave signals without attenuation. This requires further investigation.
Accuracy and range of mmWave
Commercially available FMCW mmWave sensors can be configured for specific applications. Parameters can be studied to achieve a good point cloud with sufficient resolution and accurate identification of distinct entities like pedestrians and bikes. The performance of mmWave can be compared to existing crowd sensing mechanisms.
Resilience to weather conditions
mmWave technology is expected to be resilient to weather conditions. This requires collecting data under varying temperature, humidity, and lighting conditions through outdoor deployment of mmWave sensors for a reasonable duration.
Confirmation of privacy
mmWave technology is privacy-friendly, but increasing resolution may decrease range. A pilot installation can confirm privacy is sufficiently guaranteed under installation conditions. It is essential to conduct the pilot following GDPR guidelines, ensuring personal data is shared only with necessary parties, not communicated over insecure networks or stored outside the European region. Participants should have the choice not to participate, and all personal data and copies should be deleted.
As a continuation of the mmWave project, we have installed three sensors on the Marineterrein to improve the mmWave sensor as a fully-fledged, privacy-friendly alternative for monitoring crowd movements. For more information, click here.
Future Directions for mmWave Technology
mmWave technology holds a promise of being a privacy-friendly crowd sensing technology. Further investigation is required to adapt the technology for outdoor usage and prove its capability as an all-weather alternative to existing techniques.
References
[1] “Dangerous e-bikes to be banned from cycle lanes,” [Online]. Available: https://www.newstalk.com/news/too-dangerous-e-bikes-to-be-banned-from-cycle-lanes-1385420.
[2] “Why the City of Amsterdam developed its own crowd monitoring technology,” [Online]. Available: https://www.itu.int/hub/2021/10/why-the-city-of-amsterdam-developed-its-own-crowd-monitoring-technology/.
[3] HIKVision, “People counting,” [Online]. Available: https://www.hikvision.com/en/solutions/solutions-by-function/people-counting/.
[4] Canon, “Crowd Counting Technology,” [Online]. Available: https://global.canon/en/technology/count2019.html.
[5] P. Ren, W. Fang and S. Djahel, “A novel YOLO-Based real-time people counting approach”.
[6] J. K. D. Lagman, A. B. Evangelista and C. C. Paglinawan, “Unmanned Aerial Vehicle with Human Detection and People Counter Using YOLO v5 and Thermal Camera for Search Operations,” in 2022 IEEE International Conference on Automatic Control and Intelligent Systems (I2CACIS), 2022.
[7] J. Weppner and P. Lukowicz, “Bluetooth based collaborative crowd density estimation with mobile phones,” in 2013 IEEE International Conference on Pervasive Computing and Communications.
[8] J. Lau, “Google Maps 101: How AI helps predict traffic and determine routes,” [Online]. Available: https://blog.google/products/maps/google-maps-101-how-ai-helps-predict-traffic-and-determine-routes/.
[9] “Principles relating to processing of personal data,” [Online]. Available: https://gdpr-info.eu/art-5-gdpr/.
[10] Xovis, “People counting sensors,” [Online]. Available: https://www.xovis.com/technology/sensor.
[11] F. Pittaluga and S. J. Koppal, “Privacy preserving optics for miniature vision sensors,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[12] Conmark, “Smart city catalogue,” [Online]. Available: https://www.comarkud.it/wp-content/uploads/2022/03/SMART-CITY-CATALOGUE_EN_low.pdf.
[13] S. Denis, B. Bellekens, A. Kaya, R. Berkvens and M. Weyn, “Large-scale crowd analysis through the use of passive radio sensing networks,” Sensors.
[14] “What is Lidar data and where can I download,” [Online]. Available: https://www.usgs.gov/faqs/what-lidar-data-and-where-can-i-download-it.
[15] S. Rangwala, “Smart Cities - Is LiDAR Making Them Smarter?,” [Online]. Available: https://www.forbes.com/sites/sabbirrangwala/2021/08/20/smart-citiesis-lidar-making-them-smarter/.
[16] C. Iovescu and S. Rao, “The fundamentals of millimeter wave radar sensors,” Texas Instruments.
[17] Texas Instruments, “Choosing 60-GHz mmWave sensors over 24-GHz to enable smarter industrial applications”.
[18] J.-H. Hoepman, “Privacy Design Strategies”.
[19] X. e. a. Yang, “MU-ID: Multi-user identification through gaits using millimeter wave radios,” in IEEE INFOCOM 2020-IEEE Conference on Computer Communications.
[20] P. e. a. Zhao, “mid: Tracking and identifying people with millimeter wave radar,” in International Conference on Distributed Computing in Sensor Systems (DCOSS).
[21] A. e. a. Ahmad, “Vital signs monitoring of multiple people using a FMCW millimeter-wave sensor,” in 2018 IEEE Radar Conference (RadarConf18).
[22] Infneon Technologies, “Health effects of mmWave radiation”.
[23] T. Wu, T. S. Rappaport and C. M. Collins, “Safe for generations to come: Considerations of safety for millimeter waves in wireless communications,” IEEE microwave magazine.