In-stargazer: Helmet-mounted Camera System for Localization in Large-scale Industrial Environments

1. Previous Research on Camera Placement and Orientation for Localization in Factories

Previous research has explored various camera placement options to improve the accuracy and usability of worker position estimation systems in factories. For example, [21] developed a system capable of real-time streaming by mounting a monocular camera on a worker’s helmet or safety glasses. However, the safety glasses weighed 312g, significantly increasing worker fatigue during prolonged use. [22] and [13] attached cameras to the back of workers’ safety vests for position estimation. [13] employed a monocular camera and an IMU-aided stereo camera, securely attaching them to the vest using a thick aluminum plate. While this configuration improved positional stability, the increased equipment weight compromised usability and worker convenience.

The orientation of the camera is another critical factor influencing the accuracy and practicality of localization systems in industrial environments. In [21], the camera was mounted on the front of the helmet or safety glasses, aligning with the worker’s field of view. This setup facilitated real-time observation but raised concerns about potential privacy violations of other workers and the exposure of sensitive equipment within secure facilities. In contrast, [22] and [13] adopted marker-based absolute positioning methods, with cameras facing downward to recognize floor-mounted markers. This downward-facing orientation enabled accurate worker position estimation but restricted the worker’s field of view and required a preparatory step of installing markers on the floor, which added complexity to the deployment process.

2. Previous Research on Image-Based Localization Challenges in Factories

Applying camera-based localization techniques in factory environments faces significant challenges due to the unique characteristics of industrial settings. Repetitive structures, frequent layout changes, and varying lighting conditions make it difficult for existing methods, such as SLAM (Simultaneous Localization and Mapping), VO (Visual Odometry), and VPR (Visual Place Recognition), to perform reliably. Additionally, research on ceiling images has revealed challenges specific to industrial applications.

Traditional localization methods, such as SLAM, VO, and VPR, face significant challenges in industrial settings. SLAM constructs maps and estimates positions but struggles with loop closure due to visually repetitive structures and frequent modifications in large-scale factories, leading to accumulated errors. VO, which estimates motion between frames without mapping, is computationally efficient but lacks error correction, causing accuracy degradation in low-texture or repetitive environments. Both methods suffer from instability in dynamic factory conditions where tracking loss is common. VPR, which relies on image matching with a pre-built database, is sensitive to viewpoint changes, lighting variations, and structural modifications. Learning-based VPR demands extensive training datasets, while non-learning approaches struggle in textureless or variable lighting conditions. These limitations undermine the reliability of traditional localization methods in industrial environments.

Ceiling images offer a stable and consistent basis for localization, as they are less affected by structural changes compared to walls or floors. Additionally, they enhance privacy by avoiding the capture of sensitive activities. Prior studies [23–26] explored ceiling-based localization but faced scalability and practicality issues. [23] employed pre-trained CNNs with remote-controlled robots for data collection, incurring high costs and inefficiencies, particularly in large-scale factories. Industrial settings with metallic structures also degraded laser-scanning accuracy due to reflections. The study’s small-scale environments (72m² and 83m², ceiling heights of 2.5m and 4m) further limited applicability. [24] utilized blueprints, but discrepancies from real-world layouts due to pipes and lighting fixtures reduced reliability. [25] focused on ceiling outline detection, which was hindered by uniform textures and obstructions, restricting its effectiveness in larger industrial spaces (tested in 2.5m × 5m areas). CV-SLAM [26] attempted natural landmark tracking but struggled with texture uniformity, structural modifications, and non-planar ceilings, increasing system maintenance complexity. In summary, previous ceiling-based localization approaches either rely on strong environmental assumptions (e.g., blueprints or distinctive ceiling structures), require costly data acquisition platforms, or have only been validated in small-scale environments. In contrast, our approach focuses on scalability and practicality in real industrial sites by combining section-based decomposition with lightweight wearable sensing.

1. Overview

Fig. 1. shows the system flow from the offline to the online process based on a monocular camera only. In the offline process, a 360° camera scans the factory interior, capturing a single video stream. The relative positions of the videos are calculated using SFM (Structure From Motion), and only ceiling portions are cropped and stored in the database. In the online process, a helmet-mounted camera captures ceiling-view images. These images are matched with the database, and the worker’s position is estimated using a PF (Particle Filter).

Fig. 1.

Scenario overview.

2. Design of a Helmet-Mounted Camera for Localization Systems

Fig. 2 is the device developed and used in this research. The monocular camera, along with the battery are housed and attached to the helmet. Once the main system is activated, the camera attached to the helmet captures an image in grayscale. This image is transmitted to the Matching System via Kafka, with a 10% compression applied for faster transmission. After compression, the image is converted into 64-bit binary data and sent using Kafka. For more details, refer to Table 1 for specifications.

Fig. 2.

Developed helmet-mounted camera.

Table 1.

Specification & functions of developed camera.

The proposed camera system is designed to minimize user inconvenience during operation while ensuring workplace security and protecting worker privacy in industrial environments. By integrating the camera into the helmet without requiring additional accessories such as vests, the system enhances usability and eliminates potential issues associated with extra equipment. Additionally, its lightweight design further reduces user discomfort, facilitating seamless integration into daily operations. The camera’s ceiling-facing orientation mitigates security risks and safeguards worker privacy, effectively addressing concerns related to workplace surveillance and personal data protection.

3. Offline Process

The process involves dividing the given space into zones and determining the minimal positions where users can exist, as well as providing the necessary database (DB) for matching. To efficiently manage the entire space, we first establish the minimal range within which users can be located and divide the entire space into grids (sub-sections). These sub-sections serve as foundational data required for detailed spatial matching tasks. Subsequently, the entire space is further subdivided into larger sections to finalize a systematic setup that enhances the overall efficiency of the system. Space decomposition process is an essential step for recognizing the environment and perform matching tasks efficiently. The construction of the DB involves four steps: 360° Video Capture, 3D Model Creation, Space Decomposition, DB Generation.

First, a 360° camera is employed to capture video of the space in which location estimation is to be performed. This serves as the initial step for creating a subsequent 3D model and extracting ceiling view images to construct the DB (database). The reasons for using a 360° camera in this research are as follows. Due to constraints such as the size of the factory, operational status, and filming schedule, repetitive and prolonged capturing may be challenging. By utilizing a 360° camera, a greater amount of information can be efficiently collected within a shorter time frame. Ceiling view images alone are insufficient for creating a 3D model and determining the positions of the captured images.

Second, a 3D model is generated using 360° images and actual spatial data, with unnecessary outlier points removed. The positional information of images is then obtained within the model. As the model reflects real spatial data, it serves as a three-dimensional map, while its top view functions as a two-dimensional map.

Third, space decomposition is utilized to divide the environment into sections and sub-sections and to store relevant information for efficient position estimation. Before space decomposition, the area is divided into sections, and criteria for user location range are established. These criteria are based on a 2D map from the 3D model, defining grids (sub-sections) of size m × n meters. Each grid represents the minimal range where a user can be positioned.

The 2D map is divided into m × n grids, with each sub-section classified by its section. Each 360° image is assigned to a sub-section based on its location and stored accordingly. Ceiling portions of the images are cropped and saved in corresponding folders. Feature points are then extracted from each image and stored in the same folder. Fig. 3 illustrates an example of Space Decomposition. In the Figure, small squares mean sub-sections, and their set is separated by sections.

Fig. 3.

Example of space decomposition.

Lastly, all the information acquired thus far is stored in a single DB (database). In this study, MariaDB was utilized for database construction. The database consists of several tables: the Factory table, which stores the factory name and the size of all sub-sections; the Section table, which contains section information; the Space table, which holds space information; the SpacePose table, responsible for managing the location information of spaces on the 2D map; the Pose table, which stores information related to each image; the Feature table, which stores the paths to the feature files of each image; and the Marker table, which stores information about access markers.

4. Online Process

The main system performs image capture and pose estimation through communication between two components: 1) Matching System, and 2) Filtering System. For communication, the middleware Kafka is utilized to exchange and process information efficiently. The camera attached to the safety helmet captures an image and sends it to the Matching System. The Matching System then estimates the position using a deep learning-based feature extractor and matcher, and transmits the results to the Filtering System. The Filtering System uses this information to optimize the position through particle filter, and sends the optimized data to the system responsible for displaying the current information.

4.1. Matching System

Fig. 4 presents an overview of the Matching System algorithm, which consists of three main stages: A) Feature Extraction and Matching, B) Fork Point Assignment, and C) Search Range Adjustment. When a query image 𝑄_𝑡 is received, the system performs pre-processing prior to feature extraction of 𝑄_𝑡. Specifically, it first checks the image quality (e.g., saturation, blur). It then verifies the camera ID and examines the image for markers. If a marker is detected, its position is retrieved from a pre-generated database and assigned as the previous position 𝑃_𝑡−1. If no marker is found, the system uses the previously stored pose corresponding to the camera ID as 𝑃_𝑡−1.

Fig. 4.

Matching system algorithm.

Next, deep learning-based feature points are extracted from the input image and matched against the pre-generated database to identify potential nearby positions. Subsequently, the system evaluates the success of the matching process and analyzes the number of matching pairs to designate a Fork Point. The Fork Point is defined as the ratio of the second-highest number of matching pairs to the highest (Second-to-First Matches Ratio, SFMR), where a higher value indicates a smaller difference between the two. That is, a smaller difference implies that the likelihoods of the probabilities of two positions are similar. In this explanation, Fork Point will be supposed as 𝑃_𝑡−2. The system determines whether the matching process was successful and estimates the displacement between the previous query image 𝑄_𝑡−1 and the current query image 𝑄_𝑡 based on feature correspondences. This displacement is then utilized to dynamically adjust the subsequent search range. The search range becomes the range that determines the surrounding positions when the feature points are matched with the surrounding positions using the DB.

A) Feature Extraction and Matching

The algorithm identifies sub-sections surrounding 𝑃_𝑡−1 by specifying a range 𝑟 (e.g. 𝑟 = 3, 5, 7 …) to define an 𝑟 × 𝑟 area centered on 𝑃_𝑡−1. Feature point information for ceiling-view images within these sub-sections is retrieved from the database to generate matching candidates. At this, segmentation of positions and feature extraction for ceiling-view images are performed during the Offline Process, specifically in the Space Decomposition and Database Generation phases.

The candidate set is constructed, feature points are extracted from the current input image 𝑄_𝑡, and image matching is performed between the received image and the candidates. If the number of matching pairs exceeds a predefined threshold, up to 𝑁 matches are stored in descending order of matching pairs. If no matches meet the threshold, the process is considered a matching failure.

B) Fork Point Assignment

If matching is successful, the system compares the highest and second highest number of matching pairs based on the SFMR (Second-to-First Matches Ratio). If the difference between these two numbers is not significant—that is, if it exceeds the SFMR threshold—the current position is designated as the Fork Point, and information about the matching results for nearby positions is stored. The reason for this approach is that matching results at the current position may be ambiguous. Therefore, if a future matching attempt fails, the system can determine that the previous Fork Point matching might have been inaccurate and reattempt matching from the Fork Point.

If matching fails, the algorithm employs the Fork Point to locate an earlier position, 𝑃_𝑡−2. At 𝑃_𝑡−2, the position with the highest number of matching pairs is $$ {\stackrel{-}{P}}_{t-1}$$, 1, that is 𝑃_𝑡−1. Since the failure occurred when the previous position was $$ {\stackrel{-}{P}}_{t-1,1}$$, the position with the secondhighest number of matching pairs at 𝑃_𝑡−2, $$ {\stackrel{-}{P}}_{t-1,2}$$, is assigned as the new 𝑃′_𝑡−1. The algorithm then generates new matching candidates as before but excludes any overlapping positions between the prior 𝑃_𝑡−1 and the new 𝑃′_𝑡−1 to enhance computational efficiency. This exclusion is based on the assumption that these positions have already been deemed invalid when the previous position was 𝑃_𝑡−1. This method is designed to efficiently overcome local minima within a short time frame. If the position candidates are determined, matching is attempted using the same method. If matching still fails, the previous position 𝑃_𝑡−1 is assigned as the current position 𝑃_𝑡.

C) Search Range Adjustment

Regardless of the initial or re-match attempt, if the match is successful, the algorithm determines the camera’s most probable position by summing the matching pairs within each sub-section, selecting the one with the highest sum as the current pose 𝑃_𝑡. If matching is successful across consecutive frames, the system considers the position estimation stabilized. At this point, the search range 𝑟 and accumulated displacement are reset to their initial states, preventing unnecessary expansion of the search area and ensuring efficient, robust localization. If matching is successful only in the current frame, only the accumulated displacement is reset. Finally, to complete the pose estimation, the positional information of surrounding candidate locations and their matching pair counts are stored as the final result and transmitted to the Filtering System.

If all re-matching attempts fail, the position result from the previous frame is stored as the position result for the current frame. Then, the displacement is calculated through feature tracking between the previous query Q_t-1 and the current query Q_t. If failures occur consecutively, this displacement accumulates. When the accumulated displacement exceeds a specific threshold, the search range r is expanded. Finally, the results (position candidates and the corresponding number of matching pairs) are sent to the Filtering System to complete pose estimation for the current frame.

This research employs SuperPoint [27] for feature extraction and LightGlue [28] for feature matching, addressing the limitations of traditional methods. SuperPoint offers fast, robust extraction and is resilient to transformations such as rotation, scale, and lighting changes. LightGlue, developed as an extension of SuperGlue [29], improves matching flexibility and computational efficiency.

Additionally, to achieve better DB-Query matching results, we rotate the query image and perform matching with the database. This is because LightGlue may struggle with matching if the angular difference between the two images is large. Therefore, the input image is rotated to 0°, 90°, 180°, and 270°, and matching is performed with each candidate. This approach leads to an increase in both the number of matching pairs and the accuracy compared to the original method. The illustration for this is shown in Fig. 5. Matching Results According to Angle.

Fig. 5.

Matching results example according to angle.

1. Setting And Explanation of Experiments

The specifications of the computer used for our experiments are as follows: CPU - Intel i7-6900K, GPU - NVIDIA RTX 4090 with 24GB VRAM, and RAM - 64GB. The experiments are conducted on the Ubuntu 20.04.6 LTS operating system.

In Offline Process, our program requires the creation of a 3D model. While there are various photogrammetry techniques for generating 3D models, we use the SFM (Structure From Motion) [30] technology to create a 3D point cloud, which is then used to generate the 3D model using Agisoft’s Metashape software (https://www.agisoft.com).

Furthermore, to evaluate the performance of our system in the Online Process, we conducted three types of experiments. We conducted an experiment to check the accuracy of our system, and compared our system’s performance with other studies. Also, we conducted an experiment to compare the results when using only the Matching System versus when incorporating the Filtering System.

In these experiments, we established two evaluation criteria for performance verification and comparison: the failure rate of pose estimation and the accuracy. The failure rate of pose estimation measures the proportion of Query images for which pose estimation was successful. It does not necessarily indicate whether the location was estimated correctly. Accuracy evaluates how precisely the pose was estimated at the section and sub-section level. This is determined by assessing the matching accuracy between the DB images of a section or sub-section and the corresponding Query frames.

To evaluate and compare the performance of our system, we conducted comparative experiments using VO (Visual Odometry) and SLAM (Simultaneous Localization and Mapping) research for image-based pose estimation. For VO, we used the deep learning-based location estimation method DPVO [31], and for SLAM, we employed the Stella VSLAM [32], which can utilize 360° panoramic images. During the comparison, VO was observed to lack an error correction mechanism for accumulated errors, and since it outputs results in a different coordinate system from ours, we adjusted the scale of the VO trajectory to match that of our system to assess accuracy.

The dataset used in our experiments is as follows. The experimental dataset consists of images captured directly from the factory, with Fig. 6. DB Images and Query Images showing examples of DB and query images used during the operation of the main system. We have three experimental datasets, which are described below. The shooting time and distance at which the datasets were captured are estimates. The total area of the space is 780m × 300m, which is divided into sub-sections of 3m × 2.5m, resulting in a total matrix of 12 × 26. Please refer to Table 2 for a more detailed explanation of the dataset.

Fig. 6.

Ceiling-view DB image(L) and Query image(R).

Table 2.

Dataset Information.

2. Result

Experimental results using VO are shown in Fig. 7. (a)-(c) illustrate VO alone, where trajectories are poorly estimated or suffer from significant error accumulation, making interpretation difficult. (d)-(f) present results after scale and direction alignment using evo (https://github.com/MichaelGrupp/evo). Despite this adjustment, the Predicted Line (Color, VO Result) still deviates in direction or fails to align with the Reference Line (Gray).

Fig. 7.

VO Result.

Stella VSLAM failed to achieve successful pose estimation. Despite attempts to create a map using 360° panoramic images and run the system with our camera’s images, pose estimation did not succeed. Analysis revealed that Stella VSLAM, which relies on the hand-crafted ORB feature, struggled to match images from different cameras, leading to failure. Using only the ceiling portion of the 360° images for mapping was also infeasible due to calibration constraints.

However, unlike other research, our method successfully estimated the position. In Table 3, the experimental results for section-level and sub-section-level evaluations conducted in the target factory are as follows. The probability of identifying the section where the user is currently located was approximately 88.2% on average. This result is attributed to the fact that the Filtering System does not immediately update the location based on the results from the Matching System. Consequently, sections that overlap during transitions contributed to this outcome.

Table 3.

Performance of the proposed method.

The Matching System provides the Filtering System with probability data across multiple candidate positions rather than a single best match, considering matching accuracy. Consequently, pinpointing the user’s exact location with 100% accuracy is challenging. Instead of instantly selecting the most probable location, the Filtering System gradually updates the position based on probabilities from other candidates. This approach may introduce slight delays between the actual and estimated positions, also affecting section determination when transitioning across boundaries.

This effect also influences pose estimation at the sub-section level. The probability of the estimated location exactly matching the user’s actual position averages 47.8%. However, considering the progressive updates of the system, the accuracy significantly improves when including cases with a one or more sub-section difference. Specifically, when accounting for both exact matches and one-sub-section differences, the location estimation accuracy reaches approximately 81.9%. Extending this to include up to two-sub-section differences further increases the accuracy to 91.4%.

Additionally, we compared localization results using only the Matching System versus combining it with the Filtering System. As shown in Table 4, incorporating the Filtering System improved the probability of correctly identifying the exact sub-section by approximately 7.3%. The probability of a one-sub-section difference remained similar or slightly increased, while the average error was comparable or slightly reduced. Despite the delay introduced by the Filtering System, it enhanced localization accuracy, leading to its integration for more reliable results.

Table 4.

Comparison of performance with or without filtering system.