Using smartphones for automation and assistance with everyday tasks has become an increasing trend. These devices have a wide range of applications, but health applications stand out as one of the most often considered fields of study. Specifically, preventive medicine and telemedicine are the most common approaches embraced with smartphones and general Internet of Things (IoT) devices [1], [2]. In the preceding decade, mobile applications ranging from RX management to complete treatment tracking [1]–[3] have been used.

One of the most challenging afflictions treated with mobile phones is anemia, a disease featuring an oligo-element need. Currently, anemia stands as the world’s second cause of malnutrition and is one of the most important public health issues globally [4]. Half of pregnant women and children under five years are affected by some type of anemia, and iron deficiency anemia (ferropenic) specifically is addressed in annual reports due to its high mortality rates in developing countries [4].

Many initiatives have been implemented over the preceding decade to help reduce the effects of anemia in developing countries [5]. Most of the causes attributed to anemia mortality involve the population’s lack of knowledge of how to identify early symptoms as well as a lack of physical home diagnostic tests, and global organizational efforts have had little success in providing medical resources and coverage to specific areas [4]. As technology has spread even faster than has medical coverage [6], this paper aims to develop a mobile application to act as a physical prevention and diagnosis tool for high-confidence hemoglobin estimation, while it is not expected to replace conventional laboratory testing any time soon, it has the advantage of performing a noninvasive analysis, which is very useful in cases of shortage of medical resources.

This article is organized as follows. In the next section presents related studies performed to date, and then continues with the description of a method of anemia detection based on the World Health Organization’s (WHO) hemoglobin color scale [7]. In testing section presents an experimental designed to field-test the method. And results section reports experimental results. Finally, the lasts sectionscontain the discussion and conclusions, respectively.

This research focuses on designing and developing an application suitable for hemoglobin estimation. First, it is important to define anemia. According to [8], anemia (ferropenic anemia) is a condition characterized by improper red cell volume and low hemoglobin concentrations. Anemia’s causes are diverse; among them are iron deficiency, blood loss or hemolysis. This disease affects almost 30 % of the world population or more than 2 billion people globally [5], [8].

According to the World Health Organization, stable hemoglobin levels are fundamental to preventing anemia. The current standards measure hemoglobin levels as grams of iron (Fe) per deciliter of blood (g/dl) [9]. [9] and [10] state that a concentration over 11 grams per deciliter (>11 g/dl) is necessary to consider an organism stable. Normally, the complete blood count (CBC) is the procedure used to obtain these results; it is the gold-standard laboratory test for analyzing blood samples focusing on hemoglobin levels, corpuscular medium volume (CMV) and the overall medical condition [4], [9]. Testing requires ordinary lab equipment and facilities, and several organizations and authors have performed important research related to strategies that can be used when medical assets are either limited or unavailable. We present those approaches below.

The World Health Organization introduced in 2002 a novel way to determine early anemia symptoms by creating a red-value color scale for blood, based on the hemoglobin levels that make blood darker in the case of high concentrations or more pallid under the opposite conditions [7]. Figure 1 shows the scale developed by the World Health Organization.

Figure 1 Figure 1 World Health Organization’s hemoglobin color scale WHO [7]

The procedure of using this scale consists of placing a minimal amount of blood on a slide and performing comparisons to determine the color group that best matches the color of the sample taken. Despite its simplicity, it has a high variability rate because of subjective visual matching, as reported by several countries’ low-end care centers [11]–[13]. [12] later concluded that confidence levels of this method relying on the human vision for group association were lower than 60 %.

However, [11] concluded that the hemoglobin color scale was suitable for performing anemia detection tests using color calibration curves when trained automated systems performed the association process. The results increased confidence levels up to 80 %.

Even though a hemogram test is a proper approach to hemoglobin assessment, some noninvasive strategies are considered if the conventional invasive test is unsuitable. The most widely used method is pulse oximetry, which uses a finger prick and pulsatile light wavelengths to determine oxygen saturation and oxyhemoglobin concentrations according to the Beer-Lambert absorbance law [14], [15] that states that every substance has the ability to absorb and radiate energy depending on its thickness, or specifically, its concentration. Such ability to retain a small amount of radiation is called absorbance; it is constant and does not depend on the radiation source (visible light is the most common source) used but is proportional to the distance light must travel into the absorbing substance. Additionally, the transmittance is defined as the quotient of the incoming luminous flow (absorbed) and the outgoing luminous flow (radiated) as shown in equation 1:

ECUACIÓN 1

is the sample’s transmittance,

is the incoming luminous flow,

is the outgoing luminous flow,

is the optical density of the measured substance, and has a value specific to every substance, and

is the sample’s absorbance, a measure inverse to transmittance.

Pulse oximetry distinguishes two types of hemoglobin: One carrying oxygen called oxyhemoglobin (HbO²) and the reduced hemoglobin (Hb). As the two types are different kinds of substances, both absorb light at different light wavelengths, giving the pulse oximeter the ability to measure the proportions of HbO² and Hb in blood flow by using two different light sources and different wavelengths [16]. The result of pulse oximetry is the estimation of oxygen saturation (SpO²) defined as the proportion of oxyhemoglobin (HbO²) in the total hemoglobin measured (Hb + HbO²) as shown in equation 2:

ECUACIÓN 2

On the other hand, spectrophotometry uses the same principle of a pulse oximeter to estimate the hemoglobin level, but the sample is exposed for a considerable amount of time to a monochrome light beam that irradiates the analysis area and uses a calibration curve based on light absorbance and transmittance of a certain substance [17]–[19]. A spectrophotometer uses the following equipment in its operation:

1. A general-purpose light source.

2. Spectrometer: a device capable of decomposing a light source into an individual monochrome beam using a prism.

3. Photodetector: a device that measures the residual radiation using a photon reader.

Figure 2 shows the process of light absorbance from source irradiation to blood and tissue absorption. In the diagram, light strikes the surface with intensity after traveling the distance without being absorbed by the air. Afterwards, light intensity decreases exponentially according to the distance of penetration into the tissue medium. Finally, light of intensity exits the absorption medium to be read by a photodetector that computes the difference ( ) between initial intensity and residual intensity .

figura 2 figura 2 Spectrophotometry workflow Wieben [16]

Even though spectrophotometry and pulse oximetry provide a consistent method for analyzing blood without venipuncture, the need for bulkier and specialized equipment remains a concern, particularly in developing countries where medical resource shortage is common.

Non-conventional approaches have been developed to address the need for specialized equipment. [20] identified the relation between conjunctive vasculature with anemia as several blood tissues irrigated the conjunctive area. As a result, the physical diagnostic of such vasculature matched with theresults determined by the WHO color scale relating the pallor of blood tissues regarding lower hemoglobin values [7] and [20]. However, this technique did not provide results with enough confidence, and another method was required, as [12], [21] and [22]observed pallor to be effective in determining early anemia symptoms and proposed examining palmar creases exposed to a high-intensity light source to verify pallor association with HCS. This process could not be automated.

[6] and [23] used the proposed method to systematically verify the consistency of palmar creases or tongue analysis and HCS and evaluated pallor to identify a consistent detection method. The findings were promising, as tests resulted in high confidence values (>80 %).

As the findings of both [6] and [23] were promising, the use of photography provided an efficient way of eliminating the human eye association factor from both invasive and noninvasive methods used at the time and prevented long exposure times to light sources that could contribute to stress for both patients and practitioners.

[24] demonstrated the use of photography to noninvasively determine hemoglobin levels using the palpebral conjunctiva. The usage of this method relied on specialized and expensive medical photographic equipment to properly analyze photographic samples. [25] provided a clear analysis of using a digital version of HCS, testing both performance and analysis capabilities . This scale provided a standard for photo-taking and comparison, aiming to automate the analysis stage in low-resource environments.

As hemoglobin color recognition is not a novel technique, the analysis and association stages have been improved using machine learning algorithms to provide higher confidence levels. [26] first used small blood samples on a plate and designed an artificial neural network to estimate hemoglobin levels. [25] also performed a similar process using the nearest neighbor algorithm.

Despite providing good methods for lower end analysis, resource mobility of health assets remains a discussion topic [9]. [3] presented a decision support system to help diagnose diabetic retinopathy using smartphones. This result showed that smartphones could effectively combine both high confidence levels and mobility.

However, the most important findings using mobile apps determining hemoglobin levels for diagnosing anemia consist of two apps. HemaApp presented by [27] is a novel mobile app using a smartphone camera and a specialized light beam on a finger prick to estimate hemoglobin levels similarly to the approach of a pulse oximeter. HemaApp, unlike others, uses a series of monochrome light time pulses and a matching algorithm to estimate the hemoglobin level based on the Beer-Lambert law feature selection [27].

The second app, the contest award-winning EyeAnemia™, takes advantage of the relationship of conjunctival vasculature and anemia detailed by [20] and, unlike HemaApp, assesses anemia but does not report a specific hemoglobin concentration [27].

Most mobile photo apps for image manipulation do not have robust cloud-based support and mostly rely on local algorithms; their processing is limited by memory and computing capabilities of the device. Moreover, light conditions have proven to be a significant challenge, particularly as several camera devices and accessories available in the market, using the same model, would be expected to produce comparable results.

In this paper, we propose an automated method for determining hemoglobin concentrations using medical smartphone photography. The application called Selfienemia initially operates under standard controlled light conditions and using the color association methods through a mathematical model generated by a cloud service consumed from a smartphone. This application uses a custom camera API outside the device’s default app to provide the maximum control of photographic variables and capture constraints. To build this application, several aspects are defined first.

The development platform for a mobile app build should be hardware-invariant to minimize the camera specification changes. This constraint is satisfied by mobile ecosystems experiencing low growth. The Microsoft® Windows Phone, and the Apple iOS ecosystems provide several hardware-invariant phone models, as the same vendor provides all handset cameras for each range. However, the Windows Phone platform was chosen over iOS to develop Selfienemia due to the following reasons:

1. It is compatible with Windows 10 devices and fully functional under the Universal Windows Platform (Windows 10 Core, IoT, and Mobile). The Universal Windows Platform (UWP) allows the source code to run on other platforms, such as IoT devices.

2. This platform provides extended image manipulation libraries in the Microsoft Imaging SDK, which is used in Microsoft® training systems to identify objects with a picture.

3. This platform can integrate App Insights, which can monitor application usage to identify improvement patterns.

4. The source code is compatible with cross-platform SDKs such as Xamarin, facilitating multiplatform availability.

5. It is comparatively easy to use RESTful services for the Web Objects store in Microsoft Azure (e.g., in the Azure Machine Learning Studio).

Photo tests must conform to common pre-established conditions where HCS has been demonstrated to be effective. According to [25], cool white fluorescent light bulbs provide a consistent color temperature for effective color association.

For this experiment, to achieve unchanged illumination, photo capture must be performed indoors using fluorescent light bulbs positioned in the center of the room. Attempting to perform the experiment outdoors may result in misleading findings.

Photo capture is device-dependent, as camera hardware tends to change over time. In this experiment, the minimal photo resolution of 72 dpi and the image size of 2 megapixels or above (2560x1920) are required.

Additionally, illumination conditions and the white balance must be adjusted digitally to attain color fidelity and consequently obtain meaningful data for color analysis. The literature suggests that color fidelity is best if the color temperature is between 5500 K and 6000 K, which corresponds to the fluorescent to natural daylight spectrum [28], [29]. The shutter speed, which is an important factor in the subject’s exposure, is set to be determined automatically, given the limitations of the SDK in adjusting this feature manually.

Furthermore, photo capture must be performed at a maximum distance of 8 cm from the subject with the phone standing 90° above the ground. Evaluated images use both tongue and conjunctiva as analysis areas [30]. Finger prick images are not considered during this stage due to the need for special equipment that increases the complete solution’s cost.

Once the application has been completely installed onto a mobile device, a color fitting model is loaded onto the device to estimate hemoglobin levels based on color components extracted from a photo. We describe the generation of this color model as follows:

1. Select a sample of at least 15 to 30 individuals. Perform an invasive hemoglobin lab test using the complete blood cell count (CBC). For this study, lab results from 34 individuals were used to perform the curve fitting. Rejection criteria for this group were the presence of conjunctivitis, skin trauma or jaundice. Additionally, pregnant applicants were not considered [26].

2. Obtain photo shots of every desired area in macro mode under conditions established in section “Illumination Conditions and Capturing Images Outdoors”. We performed this procedure thrice per individual to reduce bias in photo capture.

3. Select the mean RGB, ISO, and exposure detected among three photo shoots of each individual using the Euclidean distance algorithm [31], [32]. Using more photographic variables and histogram data helps create a better model but incurs the increased compute cost. However, in this study the computed parameters only include the ISO and the exposure control. We do not adjust other variables, such as the focal aperture, the shutter speed, and the white balance due to the unchanged illumination conditions set for this experiment. Therefore, the device determines settings for these variables automatically.

4. Perform exponential, polynomial and logarithmic curve fitting to datasets using numeric derivative-free approximations and choose the one offering the lowest residual mean.

5. Obtain polynomial, exponential and logarithmic curve fitting adjustments for both combined datasets. Select this as smartphone color model. We show the equations for the adjustments as follows:

ECUACIÓN 3

where:

is the hemoglobin concentration (g/dl) measured using the gold-standard invasive testing,

is the adjusted image color component of red, green and blue (RGB) values, and

are empirical constants estimated using a third-order trust region model, the RGB data and the CBC hemoglobin count.

In step 3, even though a color analysis must include in the sRGB spectrum measures of red, blue, green and alpha attenuations in samples under standard and controlled light conditions, the alpha measures are almost constant, so we do not consider it in curve fitting.

figura 3 figura 3 Overview of color model generation after executing equation 1, of curve fitting modelsection own elaboration

Figure 3 shows the model generation steps (2 to 5), in which outputs of several mathematical equations (one for each image analysis component) are used later by the phone to perform an autonomous analysis. This process can be automated and repeated with newly obtained data in the future using a cloud provider (such as Microsoft Azure) to generate a stronger association model and reduce the false positive detection rate.

First, to build the estimation model, the significance of approximations in step 5 of curve fitting model sectionmust be estimated. To estimate , 3 individual (patient) approximations must be computed to generate a single value (the Hb estimate).

The Microsoft® Azure Machine Learning Studio platform was used to compute parameter estimates for the approximation models presented above using two data mining techniques: a two-class logistic regression algorithm and a neural network implementation. We selected this platform due to its ability to create incremental datasets for either training or blind test and output results to a web service that can be accessed later by a mobile application (Selfienemia). Figure 4 shows in brief the workflow of the Azure ML Studio parameter computation stage. We describe the parameter estimation procedure as follows:

1. Project only columns for the blood dataset in the project.

2. Clean missing patient data as samples are obtained from various places.

3. Run the PCA analysis to include only the necessary data during training steps. This is performed to mitigate the computation of irrelevant variables in the dataset.

4. Split data for testing generated models later. The chosen proportion was 70 % for training and 30 % for testing.

5. Use a two-class logistic regression estimator and a neural network estimator on the sample computed dataset.

6. Test the model, choose the significant parameters and lower the variability of the approximation.

figura 4 figura 4 Overview of parameter estimation for approximations in curve fitting modelsection own elaboration

After testing the curve fitting models explained in section “Curve Fitting Model”, a linear fitting model was observed to best approximate the data gathered from the lab test and the phone. Since the original HCS has only a small set of red variants, we must identify at least three sub color variants to have enough data to determine a curve fitting model. We obtain the following linear model:

ECUACIÓN 4

where

is the estimated hemoglobin concentration,

are the estimated parameters,

are the RGB estimates supplied in the curve fitting step,

is the light sensitivity of the imaging device the image is captured, and

is the standard deviation of known measures in control group photos.

According to the model’s definition for sample surfaces with darker skin tones, lower red values of RGB samples result in higher Hb values.

After the computing stages, the app continues to follow the parameters by using a RESTful service. An advantage of this implementation is that the generated model can learn continuously as well as perform significance assessments as it gains extra information, e.g., the photo capture device information (i.e., aperture, focal length or exposure), in further experiments.

For testing purposes, we considered a sample of 64 patients to blind test both significance and confidence between conventional lab tests and Selfienemia’s results. Before testing, all 64 patients gave written medical consent. A complete blood cell count was performed on all selected applicants to obtain a comparison factor. Patients’ ages ranged from 18 to 67 years, and we imposed rejection criteria similar to those used in model generation. Such criteria considered the presence of conjunctivitis, skin trauma or jaundice. Additionally, this experiment excluded pregnant individuals [26]. The origin and gender characteristics of the population tested are presented in table 1.

Table 1 Table 1 Overview of ethnicity of patients chosen for this study

According to table 1, the ethnic groups in this experiment include Latinos of mixed ethnicities, Caucasians, African American individuals and subjects from several indigenous groups that exist in Colombia (identified as “Mixed”). In contrast to the model generation group, the photo-taking only included one instead of three images of every area, following the image capturing specifications in section “Selfienemia”. Both sets of samples, lab results, and images were used to create the estimation model in the cloud application.

The application of the Delta E color distance algorithm was still required to evaluate further how inclination and light incidence affected color reproduction. Finally, a persistence module was developed to describe an individual’s evolution while using the application in different conditions as well as to improve model generation. To evaluate the results obtained by the mobile application, we used a Bland-Altman agreement plot along with a conventional scatterplot focusing on the residual information and loss.

The persistence modules include both mobile support and cloud asset storage of previous user data with user consent. Data will be used for further analysis to improve the confidence in smartphone application’s results compared to those of invasive analysis. To explain the process better, figure 5 summarizes the testing workflow.

Figure 5 Figure 5 own elaboration

After the data collection according to the workflow in figure 4 had concluded, the resulting dataset for the sample group contained the following components:

1. CBC lab test’s hemoglobin levels.

2. Application-output estimated level.

3. The percentage difference between values in items 1 and 2.

4. The difference in Hemoglobin values (g/dl) between invasive laboratory testing and the application output.

5. Patient’s diagnosis according to the World Health Organization’s color scale.

The application output using the newly estimated model is shown below in figure 6.

figura 6 figura 6 Selfienemia’s test output for the tongue application own elaboration

A total of 128 images (64 tongue images and 64 conjunctiva images) were collected using the mobile application. The mean age of the evaluated patients was 22.1±18.1 (S.D.) years with a range of 18-67 years. The means and standard deviations for the invasive test (CBC), and the results obtained from the mobile application using tongue as the analysis area and the mobile application using conjunctiva are shown below in table 2.

Table 2 Table 2 Overview of patient’s results according to both invasive and noninvasive tests own elaboration

At the first glance, both the African American and mixed groups present the highest mean error in comparison with the invasive test, which suggests that skin color has some effect on the captured color, especially that in the conjunctiva test. We obtained the Pearson rank order correlations for both tongue and conjunctiva test sets, with the results of and , respectively. As to the error between the samples and conventional invasive testing, Selfienemia attained an impressive value of 0.62 g/dl among the general population using tongue as the analysis area and the error of 4.74 using the conjunctiva.

A breakdown of collected images was additionally performed to analyze how color bands performed individually, identifying healthy individuals. C

Figure 7 Figure 7 Color histograms for a healthy patient’s determination own elaboration

Figure 7 implies that the red channel is the discriminating feature when clustering individuals into “healthy” (Hb > 12 g/dl) and “unhealthy” (Hb < 12 g/dl) groups, while green and blue histograms are less discriminating. However, when we repeated this procedure for lower Hb levels such as 4, 6, 8 and 10, we observed that the green channel tended to perform better in distinguishing low results [25].

Our implementation of Selfienemia under controlled illumination conditions shows favorable comparisons to traditional invasive testing. Even though Selfienemia has achieved significant results, it still cannot replace laboratory tests such as CBC; nonetheless, it provides an efficient candidate tool for screening in cases of resource shortage. Figure 8 presents the Bland-Altman graph that shows that the mobile app achieved satisfactory performance, since there are no atypical values (outside two standard deviations). The bias turns out to be close to zero (±0.1091 g/dL). As a result, Selfienemia, under controlled illumination conditions, agrees positively with conventional lab tests.

Figure 8

(a) Bland-Altman plot of tongue images captured and CBC

(b) Comparison of lab measurements (red circles) and photo-enhanced mobile app used on tongue analysis (blue asterisks)

(c) Bland-Altman plot of conjunctiva images captured and CBC

(d) Comparison between lab measures (red circles) and photo-enhanced mobile app used on conjunctiva analysis (blue asterisks)

As a result of this study, we conclude in agreement with [3] that mobile applications can be used effectively in the medical field to create decision support systems for disease detection and diagnosis. This study achieved low-cost hemoglobin estimation using medical photography of the tongue and the conjunctiva, and artificial intelligence, which represents its most significant contribution to future research.

Hemoglobin determination via mobile applications was observed to be suitable for future development, as was the use of the application under controlled and standardized image capture conditions. Consequently, under controlled conditions, image capture and analysis provided results with high confidence levels as well as test automation.

We conclude that the sensor size is a key factor in capturing quality images, as are phone inclination and the light incidence angle. Wider aperture sensors manage to capture more light, resulting in more consistent colors and reliable images [25] .

In the future research, we need to develop an illumination-aware association algorithm accounting for device capabilities under nonstandard conditions. We propose including mobile sensor device information for both standardized and nonstandard image capture conditions to reduce phone resource usage and represent hardware variability.