• Current APS diagnosis often requires the cessation of anticoagulation therapy in patients who experienced thrombosis.

  • An NN, based on TG-derived parameters, diagnoses APS in anticoagulated patients with an accuracy of 92%.

Abstract

Thrombosis is an important manifestation of the antiphospholipid syndrome (APS). The thrombin generation (TG) test is a global hemostasis assay, and increased TG is associated with thrombosis. APS is currently diagnosed based on clinical and laboratory criteria, the latter defined as anti-cardiolipin, anti–β2-glycoprotein I antibodies, or lupus anticoagulant (LA). APS testing is often performed after a thrombotic episode and subsequent administration of anticoagulation, which might hamper the interpretation of clotting assays used for LA testing. We set out to develop an artificial neural network (NN) that can diagnose APS in patients who underwent vitamin K antagonist (VKA) treatment, based on TG test results. Five NNs were trained to diagnose APS in 48 VKA-treated patients with APS and 64 VKA-treated controls, using TG and thrombin dynamics parameters as inputs. The 2 best-performing NNs were selected (accuracy, 96%; sensitivity, 96%-98%; and specificity, 95%-97%) and further validated in an independent cohort of VKA-anticoagulated patients with APS (n = 33) and controls (n = 62). Independent clinical validation favored 1 of the 2 selected NNs, with a sensitivity of 88% and a specificity of 94% for the diagnosis of APS. In conclusion, the combined use of TG and NN methodology allowed for us to develop an NN that diagnoses APS with an accuracy of 92% in individuals with VKA anticoagulation (n = 95). After further clinical validation, the NN could serve as a screening and diagnostic tool for patients with thrombosis, especially because there is no need to interrupt anticoagulant therapy.

Antiphospholipid syndrome (APS) is an autoimmune disorder characterized by the presence of anti-phospholipid antibodies (aPL) and thrombosis and/or pregnancy morbidity.1,2 Due to unclarities in the pathophysiological background of APS, the syndrome is currently classified based on a combination of laboratory and clinical criteria.3,4 The clinical and laboratory criteria are defined as vascular thrombosis or pregnancy morbidity and the presence of anti-cardiolipin (aCL) antibodies and/or anti–β2-glycoprotein I (aβ2-GPI) antibodies and/or lupus anticoagulant (LA), respectively.3 Patients who display at least 1 laboratory and 1 clinical criterion are diagnosed with APS. Multiple assays are available to measure LA, aCL, and aβ2-GPI antibodies.5-7 Because the diagnostic criteria depend in part on laboratory measurements, dissonance in test results between hospitals and manufacturers may result in a different diagnosis for the same patient.6-9 

We have previously used a combination of global thrombosis tests and artificial intelligence in a clinical symptom–driven diagnostic approach. In this approach, laboratory testing for a prothrombotic phenotype was performed using the thrombin generation (TG) test.10 The TG test is known to correlate with thrombosis11,12 and differ between patients with APS and healthy individuals.13-17 Thrombin dynamics (TD) analysis quantifies the procoagulant and anticoagulant processes that govern TG.18 TD analysis has shown that prothrombin conversion (PC) and thrombin inactivation are significantly altered in APS.10,13 The accelerated PC indicates a prothrombotic state in patients with APS.13 Additionally, the activated protein C pathway is less effective in patients with APS.14-16 

Even though significant differences exist between controls and patients with APS in the dynamics of TG, it is not possible to distinguish between these populations based on 1 single test.13 We previously used the artificial intelligence approach, neural networking, to develop a model that diagnoses patients with APS from a control cohort.10 An artificial neural network is a computational model inspired by the structure of the human brain and consists of a collection of artificial neurons.19 It can be trained to classify patients based on a database of characteristics of each patient, such as general patient information and laboratory test results. The previous neural network was trained to distinguish between healthy individuals and patients with APS based on TG and TD data, that is, to diagnose APS.10,13 Importantly, in this initial project, the neural network was developed for deployment on nonanticoagulated patients and controls.10 This excludes a substantial part of patients with suspected APS, because most patients receive anticoagulant treatment immediately upon the development of a thrombotic event.20 Vitamin K antagonists (VKAs) are the recommended treatment for thrombotic APS, although direct oral anticoagulants can be considered in some situations. In this study, we focused on patients with thrombosis treated with VKAs.21 Due to LA testing being based on coagulation assays, interpretation of test results might be difficult in patients during VKA treatment. Therefore, guidelines advise to test for LA whenever possible before starting anticoagulation or to interrupt anticoagulant therapy.22-24 Because VKA discontinuation, even for a short time, is far from ideal for patients at high risk for thrombosis, we set out to develop an APS diagnosing neural network that does not require the temporary cessation of VKA anticoagulant therapy, based on the results of TG and TD assays.

Patients and sample collection

The first set of plasma samples was collected at the University Hospital of Nancy (France). The second set of plasma samples was a subset of an APS multicenter study for which patients and controls were enrolled at multiple European medical centers.6 A total of 48 and 33 patients with APS (obstetric or thrombotic) on VKA therapy and 64 and 62 control participants on VKA therapy were enrolled in the developmental (Nancy) and clinical validation (multicenter study) cohorts, respectively. In addition, 38 thrombosis controls were enrolled in the study, who were comparable in age with the patients with APS (47 years on average) and predominantly female (58%). All thrombosis controls tested negative for LA, aCL, and aβ2-GPI antibodies, and 2 patients were on VKA therapy.

All patients with APS were diagnosed according to the Sydney criteria, and laboratory tests for aPL were performed according to the International Society on Thrombosis and Haemostasis-Scientific and Standardization Committee criteria; all patients were on VKA therapy (Table 1).22,25 The research was approved by the institutional review boards and the local ethics committees of the participating centers. The study was performed in accordance with the Declaration of Helsinki. Age <18 years was an exclusion criterion for all groups. Blood was collected on 3.2% citrate in a 9:1 ratio, and platelet poor plasma was prepared by double centrifugation.24 

Table 1.

General data of the participants in the developmental and clinical validation cohorts

Developmental cohortsClinical validation cohorts
Patients with APS (n = 48)Controls (n = 64)Patients with APS (n = 33)Controls (n = 62)Thrombosis controls (n = 38)
Age, mean ± SD, y 46 ± 14 67 ± 11  47 ± 12 69 ± 11  47 ± 13  
Sex, n (% male) 18 (38) 47 (73)  9 (27) 51 (82)  16 (42)  
Primary APS, n (%) 30 (62)  33 (100)   
Secondary APS, n (%) 18 (38)  0 (0)   
LA, n (%) 39 (81)  29 (88)  0 (0) 
Anti-CL antibodies, n (%) 38 (79)  25 (76)  0 (0) 
Anti–β2-GPI antibodies, n (%) 27 (56)  34 (81)  0 (0) 
Thrombosis, n (%) 48 (100) 4 (6%)  31 (94) 7 (11)  38 (100)  
Pregnancy morbidity, n (%) 8 (17)  4 (12)  0 (0) 
INR
(treatment range21,26,27
NA
(2.0-3.0) 
2.8 ± 0.8
(2.0-3.0) 
2.5 ± 0.9
(2.0-3.0) 
3.1 ± 1.0
(2.0-3.0) 
2.1 ± 0.3 (VKA)
1.1 ± 0.3 (non-VKA)
(2.0-3.0) 
Developmental cohortsClinical validation cohorts
Patients with APS (n = 48)Controls (n = 64)Patients with APS (n = 33)Controls (n = 62)Thrombosis controls (n = 38)
Age, mean ± SD, y 46 ± 14 67 ± 11  47 ± 12 69 ± 11  47 ± 13  
Sex, n (% male) 18 (38) 47 (73)  9 (27) 51 (82)  16 (42)  
Primary APS, n (%) 30 (62)  33 (100)   
Secondary APS, n (%) 18 (38)  0 (0)   
LA, n (%) 39 (81)  29 (88)  0 (0) 
Anti-CL antibodies, n (%) 38 (79)  25 (76)  0 (0) 
Anti–β2-GPI antibodies, n (%) 27 (56)  34 (81)  0 (0) 
Thrombosis, n (%) 48 (100) 4 (6%)  31 (94) 7 (11)  38 (100)  
Pregnancy morbidity, n (%) 8 (17)  4 (12)  0 (0) 
INR
(treatment range21,26,27
NA
(2.0-3.0) 
2.8 ± 0.8
(2.0-3.0) 
2.5 ± 0.9
(2.0-3.0) 
3.1 ± 1.0
(2.0-3.0) 
2.1 ± 0.3 (VKA)
1.1 ± 0.3 (non-VKA)
(2.0-3.0) 

INR, International normalized ratio; NA, not available; SD, standard deviation.

P < .001 compared with patients with APS.

No significant difference compared with APS patients.

INR data were not available for all participants; results for a subset of the cohorts is shown.

APS-specific testing

aPL positivity was defined as positivity for LA and/or aCL antibodies and/or aβ2-GPI immunoglobulin G/immunoglobulin M.3 LA positivity was determined by each local center, according to the International Society of Thrombosis and Haemostasis-Scientific and Standardization Subcommittee guideline.28 In the first APS cohort, aCL antibody presence was tested by enzyme-linked immunosorbent assay,29 and aβ2-GPI antibodies were detected as previously described.13,30 In the second APS and non-APS thrombotic cohorts, aCL and aβ2-GPI antibodies were detected using the QUANTA Lite enzyme-linked immunosorbent assay (Inova Diagnostics) according to the manufacturer’s specifications.

Coagulation factor measurements

Plasma fibrinogen levels were measured by the Clauss method on the STA-R analyzer according to the manufacturer’s instructions (Diagnostica Stago, Asnières-sur-Seine, France). Functional antithrombin (AT) levels were determined using a chromogenic substrate on the STA-R analyzer according to the manufacturer’s instructions (Diagnostica Stago). α2-macroglobulin (α2M) was measured in a functional home-made assay (Synapse Research Institute, Maastricht, The Netherlands).18 

TG

TG was measured by calibrated automated thrombinography using PPP reagent low and PPP reagent (Diagnostica Stago). TG was measured in the presence or absence of thrombomodulin (TM) to test the sensitivity of the activated protein C system (Synapse Research Institute).31 The concentration of TM was titrated to achieve a 50% inhibition of the ETP in pooled normal plasma.31 Pooled normal plasma was prepared locally from plasma obtained from 40 healthy donors.

TD

The TD method was used to calculate PC and thrombin inactivation parameters from TG data, as described in detail elsewhere.18 The rate of thrombin inactivation was computed based on plasma AT, α2M, and fibrinogen levels. PC curves were calculated from TG curves using the computed thrombin inactivation rate.

The generated PC curve was quantified by the total amount of prothrombin converted (PCtot) and the maximum PC rate (PCmax). Additionally, the formation of thrombin-AT (T-AT) and thrombin-α2M (T-α2M) complexes was obtained from TD analysis. The thrombin decay constant was determined separately, which is defined as the overall capacity of a plasma sample to inactivate thrombin.

Neural network development

Neural networks were trained to diagnose patients with APS from a cohort of anticoagulated individuals using Matlab’s Neural Net Pattern Recognition Application (Neural Network Toolbox, version 9.0.0.341360 R2016a; Mathworks, Natick, MA). Neural networks are so-called black box models because researchers cannot define the way how individual artificial neurons work together to arrive at the final output, although some modulation is possible.

In this study, neural networks were structured as 2-layer feed-forward networks, with sigmoid hidden and softmax output neurons, and containing 10 neurons in the hidden layer. Neural networks were trained with scaled conjugate gradient backpropagation. Training, validation, and testing data sets contained 70%, 15%, and 15% of the samples, respectively, and the samples were randomly assigned to 1 of 3 data sets before every neural network run. Owing to the random assignment of samples to the training, validation, or testing data sets before each neural network run, differences exist between each developed neural network.

The input of the APS-diagnosing artificial neural network consisted of TG parameters (lag time, time-to-peak, peak height, ETP, and velocity index), TD parameters (PCtot, PCmax, thrombin-AT complexes, T-α2M complexes, and thrombin decay constant), and the inhibition of peak height and PCmax by TM. The outputs of the neural networks were either “APS patient” or “control subject.” Neural network training was performed 5 consecutive times using the developmental cohort database (48 patients with APS and 64 anticoagulated controls). The accuracy of each developed neural network was quantified using ROC analysis (sensitivity and specificity) and by calculating the positive and negative predictive values. When the neural network was applied to new clinical samples, each sample was appointed a value between 0% and 100%, representing the likelihood of the sample being a control sample (closer to 0%) or a patient with APS sample (closer to 100%).

The 2 neural networks with the highest performance were selected for clinical validation in the separate clinical validation cohort (33 APS patients and 62 anticoagulated controls). From this analysis, the best-performing neural network was selected, and its accuracy was further tested in patients with APS of the validation cohort (n = 62) and the non-APS thrombosis controls (n = 38).

Statistical analysis

Statistical analyses were performed in GraphPad Prism (version 10.0.1; GraphPad Software, Boston, MA). The Mann-Whitney test or the Student t test was used according to the distribution of data to compare differences between the groups, and statistical significance was reported as P values <.05.

For the developmental cohort, the hemostatic profile was assessed in 48 patients with APS and 64 controls, both on VKA treatment. Patients with APS were significantly younger and more often female than control participants (P < .001 for both; Table 1 “Developmental cohort”). Sixty-two percent of patients with APS suffered from primary APS, whereas 38% suffered from secondary APS (all patients suffered from lupus). LA was detected in 39 patients with APS (81%), whereas aCL and aβ2-GPI antibodies were detected in 38 (79%) and 27 patients (56%) with APS, respectively. All patients with APS experienced prior thrombosis, compared with 6% of the anticoagulated control population, and 17% of patients with APS suffered from pregnancy morbidity.

The hemostatic system of patients with APS and controls was studied by TG and TD analysis (Figure 1). TG was used to investigate the potential of each patient to generate a procoagulant TG response, regardless of their anticoagulant therapy.

Figure 1.

TG and TD parameters in patients with APS (n = 48) and control participants (n = 64) in the neural network developmental cohort. (A-C) TG parameters lag time (A), peak height (B), and time-to-peak (C) did not differ significantly between patients with APS and controls on VKA treatment. (D-E) The ETP (D) was significantly lower in patients with APS than controls; whereas the velocity index (E) was significantly higher. (F) The inhibitory effect of TM on the peak height was significantly less in patients with APS compared with controls. (G-K) TD parameters PCtot (G), thrombin-AT (T-AT) complex formation (J), T-α2M formation (K) were comparable between anticoagulated patients with APS and controls; (H, I) PCmax was significantly increased in patients with APS (H), and the PCmax was less sensitive to the inhibitory actions of TM (I). (L) The thrombin decay capacity was significantly higher in patients with APS than controls. Differences between group means were analyzed using the Mann-Whitney test. ∗P < .05; ∗∗∗P < .001; ∗∗∗∗P < .0001. Data are shown as median and interquartile ranges.

Figure 1.

TG and TD parameters in patients with APS (n = 48) and control participants (n = 64) in the neural network developmental cohort. (A-C) TG parameters lag time (A), peak height (B), and time-to-peak (C) did not differ significantly between patients with APS and controls on VKA treatment. (D-E) The ETP (D) was significantly lower in patients with APS than controls; whereas the velocity index (E) was significantly higher. (F) The inhibitory effect of TM on the peak height was significantly less in patients with APS compared with controls. (G-K) TD parameters PCtot (G), thrombin-AT (T-AT) complex formation (J), T-α2M formation (K) were comparable between anticoagulated patients with APS and controls; (H, I) PCmax was significantly increased in patients with APS (H), and the PCmax was less sensitive to the inhibitory actions of TM (I). (L) The thrombin decay capacity was significantly higher in patients with APS than controls. Differences between group means were analyzed using the Mann-Whitney test. ∗P < .05; ∗∗∗P < .001; ∗∗∗∗P < .0001. Data are shown as median and interquartile ranges.

Close modal

Figure 1 shows that although patients with APS on average have a significantly lower ETP than controls (P = .011; Figure 1D), the velocity index, that is, the rate at which the burst of TG occurs, was significantly higher on average (P < .0001; Figure 1E). Moreover, TG in patients with APS was significantly less sensitive to the anticoagulant action of TM on average (P < .0001; Figure 1F). The TG peak height and the time-dependent parameters, lag time and time-to-peak, did not differ significantly between patients and controls (Figure 1A-C). However, the overlap between the patient and control groups does not allow the distinction between normal controls and patients with APS based on single TG parameters.

TD analysis was used to examine the balance between PC and thrombin inactivation in patients with APS and controls. PCmax was significantly increased on average in patients with APS compared with controls (P = .0263), whereas PCtot was comparable (Figure 1G-H). PCmax was less sensitive to the inhibitory actions of TM in patients with APS on average (P < .0001; Figure 1I), in line with the reduced effect of TM on TG. Thrombin inhibitor complex formation was comparable between controls and patients with APS (Figure 1J-K). Nevertheless, the average intrinsic thrombin inhibitory capacity of patients with APS and control participants, that is, the thrombin decay capacity, was higher in patients with APS than in controls (P = .0004; Figure 1L).

Similar to TG parameters, the overlap between the patient and control groups does not allow for the distinction between normal controls and patients with APS based on single TD parameters. Therefore, we used the artificial intelligence method of neural networks that incorporate all TG and TD parameters into 1 prediction model to diagnose anticoagulated patients with APS from a group of anticoagulated controls.

Five independent neural networks were developed, using 6 TG and 6 TD parameters as inputs. The performance of each neural network throughout the developmental phase are shown in detail in supplemental Figure 1. Neural network 1 (NN01) and NN03 were the most accurate neural networks for the diagnosis of APS, because they correctly classified 61 of 64 controls (95%) and 47 of 48 patients with APS (98%); and 62 of 64 controls (97%) and 46 of 48 patients with APS (96%), respectively. The least accurate neural network was NN04, which correctly classified 60 of 64 controls (94%) and 44 of 48 patients with APS (92%).

The performance of the 5 developed neural networks was determined as the overall accuracy, positive and negative predictive value, and specificity and sensitivity (Table 2). NN01 showed the highest negative predictive value (98.4%) and sensitivity (97.9%); whereas NN03 showed the highest positive predictive value (95.8%) and specificity (96.9%). Both NN01 and NN03 had an identical overall accuracy (96.4%), and the performances of NN02, NN04, and NN05 were inferior compared with NN01 and NN03. Therefore, NN01 and NN03 were selected for an in-depth performance analysis and clinical validation (Figure 2).

Table 2.

Quantification of the performance of the 5 developed neural networks in the developmental phase

NN01NN02NN03NN04NN05
Positive predictive value (%) 94.0 95.6 95.8 91.3 93.6 
Negative predictive value (%) 98.4 92.5 96.9 90.9 93.8 
Sensitivity (%) 97.9 89.6 95.8 87.5 91.7 
Specificity (%) 95.3 96.9 96.9 93.8 95.3 
Overall accuracy (%) 96.4 93.8 96.4 91.1 93.8 
NN01NN02NN03NN04NN05
Positive predictive value (%) 94.0 95.6 95.8 91.3 93.6 
Negative predictive value (%) 98.4 92.5 96.9 90.9 93.8 
Sensitivity (%) 97.9 89.6 95.8 87.5 91.7 
Specificity (%) 95.3 96.9 96.9 93.8 95.3 
Overall accuracy (%) 96.4 93.8 96.4 91.1 93.8 

The positive and negative predictive values, sensitivity and specificity, and the overall accuracy were determined for all 5 neural networks (NN01-NN05) in the developmental cohort during the developmental phase and in a separate clinical validation cohort in the clinical validation phase. Overall, NN01 and NN03 showed the most accurate performance.

Figure 2.

Performance of neural networks NN01 and NN03 for the differentiation between patients with APS and controls treated with VKAs. (A-B) Confusion matrices showing the accuracy of patients with APS classification by NN01 (A) and NN03 (B) in the developmental phase. (C-D) ROC curve of the diagnostic accuracy of NN01 (C) and NN03 (D) to identify patients with APS. (E-F) Quantification of APS suspicion by NN01 (E) and NN03 (F) according to subject type. (G) Quantified sensitivity and specificity of NN01 (blue) and NN03 (green) for the diagnosis of APS. (H) Positive and negative predictive values of NN01 (blue) and NN03 (green) for the diagnosis of APS.

Figure 2.

Performance of neural networks NN01 and NN03 for the differentiation between patients with APS and controls treated with VKAs. (A-B) Confusion matrices showing the accuracy of patients with APS classification by NN01 (A) and NN03 (B) in the developmental phase. (C-D) ROC curve of the diagnostic accuracy of NN01 (C) and NN03 (D) to identify patients with APS. (E-F) Quantification of APS suspicion by NN01 (E) and NN03 (F) according to subject type. (G) Quantified sensitivity and specificity of NN01 (blue) and NN03 (green) for the diagnosis of APS. (H) Positive and negative predictive values of NN01 (blue) and NN03 (green) for the diagnosis of APS.

Close modal

Figure 2A-B show the comparison of the confusion matrices of the development of NN01 and NN03. ROC curve analysis showed a high sensitivity and specificity for both neural networks, although NN03 was slightly more accurate (ROCAUC, 0.9941; confidence interval [CI], 0.9847-1.000) than NN01 (ROCAUC, 0.9857; CI, 0.9647-1.000). The ROCAUC values were higher for both NN01 and NN03 than for individual variables when used separately for the diagnosis of APS. The ROCAUC for the individual variables ranged from 0.5003 (CI, 0.3903-0.6104) for the peak height to 0.86 (CI, 0.7911-0.9315) for the inhibitory effect of TM on the peak height (supplemental Table 1; supplemental Figure 2).

The neural network expresses the suspicion of APS as a value between 0% and 100%. A value close to 0% indicates a low chance of APS, and a percentage close to 100% indicates a high likelihood of APS. Figure 2E-F show that both in NN01 (Figure 2E) and NN03 (Figure 2F), the average APS suspicion values of the control group are 11.1% ± 17.8% and 6.4% ± 15.0%, respectively, whereas the scores are 91.5% ± 17.6% and 95.3% ± 16.9% in the APS group. Figure 2G-H show that NN01 scored the highest on negative predictive value (98.4%) and sensitivity (97.9%), whereas NN03 had the highest accuracy in positive predictive value (95.8%) and specificity (96.9%).

Clinical validation

The clinical validation cohort consisted of 33 patients with APS and 62 control participants, both on VKA treatment (Table 1 “Clinical validation cohort”). Patients with APS were significantly younger than control participants (average age, 47 vs 69 years; P < .001) and more predominantly female (73% vs 18%; P < .0001). Ninety-four percent of patients with APS suffered from prior thrombotic events, compared with 13% of the control participants (P < .001). LA was detected in 88% of patients with APS, and aCL antibodies were present in 76% and aβ2-GPI antibodies in 81% of patients with APS.

TG and TD were measured to establish a separate validation cohort for the APS-diagnosing neural network (Figure 3). Lag time (Figure 3A) and time-to-peak (Figure 3C) were significantly prolonged in patients with APS. The peak height (Figure 3B), ETP (Figure 3D), and velocity index (Figure 3E) were reduced in patients with APS compared with controls. Moreover, the peak height in patients with APS was significantly less sensitive to TM compared with controls (Figure 3F). The lower TG in patients with APS can be explained by a lower PCtot (Figure 3G), a lower PCmax (Figure 3H), and a higher thrombin inactivation capacity (Figure 3L). Additionally, the process of PC showed a marked TM resistance in patients with APS (Figure 3I).

Figure 3.

TG and TD parameters in the patients with APS (n = 33) and control participants (n = 62) in the neural network clinical validation cohort. (A-F) The TG lag time (A) and time-to-peak (C) were significantly prolonged in patients with APS, whereas the peak height (B), ETP (D), velocity index (E), and the effect of TM on the peak height were reduced (F); and time-to-peak (C) did not differ significantly between patients with APS and controls on VKA treatment; the ETP (D) was significantly lower in patients with APS than controls, whereas the velocity index (E) was significantly higher. (F) The inhibitory effect of TM was significantly less in patients with APS compared with controls. (G-K) TD parameters PCtot (G), T-AT complex formation (J), and T-α2M formation (K) were significantly lower in patients with APS. (H) PCmax was significantly decreased in patients with APS, and the PCmax was less sensitive to the inhibitory actions of TM (I). (L) The thrombin decay capacity was significantly higher in patients with APS than controls. Differences between group means were analyzed using the Mann-Whitney test. ∗P < .05; ∗∗∗P < .001; ∗∗∗∗P < .0001. Data are shown as median and interquartile ranges.

Figure 3.

TG and TD parameters in the patients with APS (n = 33) and control participants (n = 62) in the neural network clinical validation cohort. (A-F) The TG lag time (A) and time-to-peak (C) were significantly prolonged in patients with APS, whereas the peak height (B), ETP (D), velocity index (E), and the effect of TM on the peak height were reduced (F); and time-to-peak (C) did not differ significantly between patients with APS and controls on VKA treatment; the ETP (D) was significantly lower in patients with APS than controls, whereas the velocity index (E) was significantly higher. (F) The inhibitory effect of TM was significantly less in patients with APS compared with controls. (G-K) TD parameters PCtot (G), T-AT complex formation (J), and T-α2M formation (K) were significantly lower in patients with APS. (H) PCmax was significantly decreased in patients with APS, and the PCmax was less sensitive to the inhibitory actions of TM (I). (L) The thrombin decay capacity was significantly higher in patients with APS than controls. Differences between group means were analyzed using the Mann-Whitney test. ∗P < .05; ∗∗∗P < .001; ∗∗∗∗P < .0001. Data are shown as median and interquartile ranges.

Close modal

TG and TD data obtained in the clinical validation cohort were used to clinically validate the 2 top-performing neural networks, NN01 and NN03 (Table 3). Although NN01 and NN03 showed comparable performance accuracies in the developmental cohort, NN01 showed a better performance in the clinical validation phase. NN01 showed an overall accuracy of 91.6% for the diagnosis of APS from a group of anticoagulated control participants. The positive and negative predictive values of the neural network were 87.9% and 93.5%, respectively. The sensitivity of NN01 was 87.9%, and the specificity was 93.5%. As expected for the performance results of the developmental phase, NN02, NN04, and NN05 were inferior in performance to NN01.

Table 3.

The quantification of performance of neural networks NN01 and NN03 in the clinical validation phase

NN01NN03
Positive predictive value (%) 87.9 85.3 
Negative predictive value (%) 93.5 93.4 
Sensitivity (%) 87.9 87.9 
Specificity (%) 93.5 91.9 
Overall accuracy (%) 91.6 90.5 
NN01NN03
Positive predictive value (%) 87.9 85.3 
Negative predictive value (%) 93.5 93.4 
Sensitivity (%) 87.9 87.9 
Specificity (%) 93.5 91.9 
Overall accuracy (%) 91.6 90.5 

The positive and negative predictive values, sensitivity and specificity, and the overall accuracy were determined for the neural networks NN01 and NN03 in the clinical validation cohort in the clinical validation phase. Overall, NN01 showed the most accurate performance.

A limitation of the currently developed neural network could be that it is trained to distinguish between patients with APS and controls on VKA therapy. However, VKA therapy is not only used in patients who experienced thrombosis, but also, for example, in patients suffering from atrial fibrillation. Subsequently, a history of thrombosis was uncommon in the VKA therapy control groups of the developmental and clinical validation cohorts (1% and 13%, respectively).

However, the suspicion of APS often arises when patients suffer from a thrombotic event, which requires the administration of anticoagulant therapy. Therefore, we analyzed a second non-APS control group consisting of patients who experienced thrombosis before our study (Table 1).

Figure 4 gives an overview of the performance of NN01 in the classification of patients into patients with APS or control participants (Figure 4A) and a population of thrombosis control participants (Figure 4B). NN01 correctly classified 29 of 33 patients with APS, 58 of 62 control participants, and 28 of 38 thrombosis control participants. ROC analysis was used to determine the diagnostic accuracy of the neural network. The ROCAUC was 0.93 (CI, 0.87-1.00) for the detection of patients with APS from a control cohort. For individual variables in the clinical validation cohort, the ROCAUC varied from 0.67 (CI, 0.54-0.79) for ETP to 0.95 (CI, 0.92-1.00) for T-α2M complexes (supplemental Table 1; supplemental Figure 2). For the thrombosis control cohort, the ROCAUC was 0.86 (CI, 0.77-0.95) for the detection of patients with APS from a thrombosis control cohort (Figure 4C-D).

Figure 4.

The validation of neural network NN01 in patients with APS (n = 33), controls (n = 62), and thrombosis controls (n = 38). (A-B) Confusion matrices showing the accuracy of diagnosis in patients with APS compared with an anticoagulated control cohort (A) and a thrombosis control cohort (B). (C-D) ROC curves of the diagnostic accuracy of NN01 to identify patients with APS from an anticoagulated control cohort (C) and a thrombosis control cohort (D). (E-F) Quantification of APS suspicion by NN01 according to subject type in anticoagulated controls (E) and thrombosis controls (F). (G) Quantified sensitivity and specificity of NN01 for the diagnosis of APS. (H) Positive and negative predictive values of NN01.

Figure 4.

The validation of neural network NN01 in patients with APS (n = 33), controls (n = 62), and thrombosis controls (n = 38). (A-B) Confusion matrices showing the accuracy of diagnosis in patients with APS compared with an anticoagulated control cohort (A) and a thrombosis control cohort (B). (C-D) ROC curves of the diagnostic accuracy of NN01 to identify patients with APS from an anticoagulated control cohort (C) and a thrombosis control cohort (D). (E-F) Quantification of APS suspicion by NN01 according to subject type in anticoagulated controls (E) and thrombosis controls (F). (G) Quantified sensitivity and specificity of NN01 for the diagnosis of APS. (H) Positive and negative predictive values of NN01.

Close modal

Quantification of the average APS suspicion was significantly higher in patients with APS (85% ± 31%) than in controls (13% ± 22%; P < .0001) and thrombosis controls (27% ± 42%; P < .0001). The sensitivity of NN01 for the detection of APS was 87.9%, and the specificity was 93.5% for the control group and 73.7% for thrombosis controls (Figure 4G). The negative and positive predictive values of NN01 for the detection of APS from a control cohort were 93.5% and 87.9%, respectively, and 74.4% and 87.9% for the detection of APS in a thrombosis cohort (Figure 4H). Interestingly, there was a clear discrimination between a certain subset of thrombosis controls, who had very high APS suspicion percentage (>88%; n = 10) compared with the remainder of the group (<22%; n = 23; Figure 4F). Ten of 38 thrombosis controls were classified as patients with APS, even though aPL testing at the treating hospital did not reveal positivity for the APS diagnostic criteria. Sixty percent of the patients were male, and they were aged 51.2 years on average. One patient experienced arterial thrombosis, whereas 9 patients experienced venous thrombosis: 4 patients suffered from deep venous thrombosis, 2 from pulmonary embolism, and 3 from cerebral thrombosis. Each sample was further tested for aCL and aβ2-GPI antibodies using 4 different platforms. All tests were negative except for 1 patient who was positive on 1 of 4 platforms. Additionally, anti-PS/PT antibodies were detected in another patient.

APS is currently diagnosed based on clinical manifestations and laboratory findings.3,32 However, thrombosis is a major clinical problem in APS, and therefore, patients are frequently treated with anticoagulants for a long duration.33,34 Moreover, APS is often suspected after the occurrence of a thrombotic event, a period when patients are treated with anticoagulants.23,34 The laboratory assays that are currently used for LA, one of the aPL tested as a laboratory criteria, are hampered by the interference of anticoagulant therapy.22 Therefore, intermission of the anticoagulant therapy is one of the options for adequate diagnosis of APS in patients who experience thrombosis.23 

The TG test is a global hemostasis assay,35 and the TG profile is known to be increased in patients at risk of thrombosis.36 TG is higher in patients with APS, although APS-related changes in a single TG parameter are not sufficient for an APS diagnosis.13 In this study, we confirmed that although TG and TD parameters differ on average between patients with APS and controls, it is not possible to differentiate between patients with APS and controls based on a single TG or TD parameter due to the significant overlap in TG and TD values between patients and controls. Therefore, the application of the neural network methodology is of interest for the classification of patients with APS, because this allows to integrate all collected data and develop a neural network that can make an accurate classification of patients with APS based on the combination of TG and TD parameters.10 

Previously, we developed a neural network to diagnose APS based on the TG profile.10 This neural network was trained on participants who did not use anticoagulants. Nonetheless, both anticoagulated and nonanticoagulated patients with APS were diagnosed correctly in almost all cases. In this study, we have developed and validated a neural network that accurately diagnoses APS specifically in VKA-anticoagulated participants, based on TG-related parameters.

Neural network NN01 diagnosed anticoagulated patients with APS accurately based on TG and TD parameters in the developmental cohort (sensitivity, 98%; specificity, 95%) and the clinical validation cohort (sensitivity, 88%; specificity, 94%). The negative predictive values of 98% and 94% in the developmental and validation cohorts, respectively, indicate that NN01 precisely rules out APS in suspected participants. ROC analysis revealed that combining parameters in a neural network, results in a more accurate diagnosis than a diagnosis based on single parameters. Although some individual parameters significantly differed between patients with APS and controls, these results were not transferrable between different cohorts of patients with APS (supplemental Table 1), except for the effect of TM on TG peak height and TD PCmax.

The generalization of the neural network in a separate cohort of patients and controls is pivotal in the neural network approach. In this study, we have shown the generalizability of our neural network in a second cohort of patients with APS and controls. We validated the 2 top-performing neural networks for the diagnosis of APS in anticoagulated participants, which led to the conclusion that NN01 can accurately differentiate between patients with APS-related thrombosis and patients with thrombosis unrelated to laboratory indicators of APS. The sensitivity of NN01 for the detection of APS in a group of control participants or patients with thrombosis is comparable, whereas the specificity of the neural network is lower for thrombosis controls than normal controls. This finding indicates that hemostatic changes occur in patients suffering from APS-related thrombosis vs APS-unrelated thrombosis, which can be picked up by the neural network.37 The TG assay is used to detect hypocoagulability and hypercoagulability, and its outcome is, therefore, expected to be associated with a bleeding or thrombotic phenotype, respectively.35 Therefore, it can be assumed that the predictions of NN01 depend at least in part on the detection of APS-related hypercoagulability, because the network is based on TG and TD input parameters. Indeed, we found a significant association between a prolonged TG lag time and the APS suspicion score (supplemental Figure 3), indicating that changes in the TG profile that are associated with APS could contribute to the diagnostic neural network. Lower peak height and shorter lag time in patients with APS could be in part caused by the younger age of patients than that of controls.38 Typically, the lag time is slightly prolonged at older age, and the TG peak height is increased.38 However, patients with APS were also more resistant to the actions of TM, which usually occurs more at an older than at a younger age.39 

A limitation of this study is the relatively low number of APS samples, which can be attributed to the low prevalence of APS in the population. Additionally, because pregnancy morbidity and thrombosis are clinical diagnostic criteria for APS, the majority of patients diagnosed with APS is female and younger than patients with thrombosis due to other causes, or control patients using VKA due to other indications.

In this study, we show a proof-of-principle for the use of neural network methodology in the fields of APS and TG. For application in the clinic, this method should be further investigated and validated in a larger, preferably prospective study in patients undergoing APS diagnostic testing. Nevertheless, we believe it is important to share our findings that neural network analysis can support the classification of samples of patients with APS and normal controls based on hypercoagulability testing. This is a novel finding, because the current APS diagnosis strategy solely relies on aPL testing and clinical criteria. Moreover, the current diagnostic laboratory assays for APS have other limitations for diagnostic laboratories, such as the interassay variation in aPL antibody assays,6 the complexity of the 3-step LA assay,24 and the need to interrupt anticoagulant therapy.23 In contrast, this neural network only requires a TG assay without the need to interrupt anticoagulation. Harmonization and standardization efforts are ongoing40 to aid the implementation of TG assays into clinical laboratories. In the future, TG assay will become more available for patient management, and the current artificial intelligence–based approach should be helpful for clinical decision-making.

In conclusion, NN01 diagnoses APS with an accuracy of 92% in VKA-anticoagulated individuals. The current neural network approach could be thought of as an initial screening tool to identify patients at risk of APS, although further research is needed to achieve this goal. If a patient is considered at risk by the neural network, further laboratory testing for APS could be advised. In this analyses, we show that the neural network is quite accurate in ruling out patients who are negative for the laboratory criteria of APS (normal and thrombosis controls). Moreover, because APS is diagnosed based on both laboratory and clinical criteria, a diagnosis can only be achieved when a patient fulfills at least 1 of the clinical criteria. Therefore, this neural network could be of interest to investigate participants who are positive for laboratory criteria but lack clinical criteria. It is interesting to investigate in a prospective study setup whether these patients do meet the clinical diagnostic criteria in the future, that is, whether they experience pregnancy morbidity or thrombosis. Therefore, future prospective studies are needed to further validate NN01 for the diagnosis of APS.

Contribution: R.M.W.d.L.-K. designed the study, acquired TG and TD data, performed neural network analyses, and drafted the manuscript; D.W. and S.Z. contributed crucial patient samples and critically revised the manuscript; M.N. acquired TG data and revised the manuscript; V.R. and J.M. contributed crucial patient samples and critically revised the manuscript; P.G.d.G. critically revised the manuscript; K.M.J.D. contributed crucial patient samples, organized the multicenter study (clinical validation cohort), and critically revised the manuscript; and B.d.L. conceived the study, supervised the work, and cowrote the manuscript.

Conflict-of-interest disclosure: R.M.W.d.L.-K., M.N., P.G.d.G., and B.d.L. are employees of Synapse Research Institute, part of Diagnostica Stago SAS. The remaining authors declare no competing financial interests.

Correspondence: Romy M.W. de Laat-Kremers, Department of Data Analysis and Artificial Intelligence, Synapse Research Institute, Pastoor Habetsstraat 50, 6217KM, Maastricht 6229EV, The Netherlands; email: r.delaat@thrombin.com.

1.
Galli
M
,
Luciani
D
,
Bertolini
G
,
Barbui
T
.
Anti-beta 2-glycoprotein I, antiprothrombin antibodies, and the risk of thrombosis in the antiphospholipid syndrome
.
Blood
.
2003
;
102
(
8
):
2717
-
2723
.
2.
Palomo
I
,
Segovia
F
,
Ortega
C
,
Pierangeli
S
.
Antiphospholipid syndrome: a comprehensive review of a complex and multisystemic disease
.
Clin Exp Rheumatol
.
2009
;
27
(
4
):
668
-
677
.
3.
Miyakis
S
,
Lockshin
MD
,
Atsumi
T
, et al
.
International consensus statement on an update of the classification criteria for definite antiphospholipid syndrome (APS)
.
J Thromb Haemost
.
2006
;
4
(
2
):
295
-
306
.
4.
Barbhaiya
M
,
Zuily
S
,
Naden
R
, et al;
ACR/EULAR APS Classification Criteria Collaborators
.
2023 ACR/EULAR antiphospholipid syndrome classification criteria
.
Ann Rheum Dis
.
2023
;
82
(
10
):
1258
-
1270
.
5.
Yin
D
,
Chayoua
W
,
Kelchtermans
H
, et al
.
Detection of anti-domain I antibodies by chemiluminescence enables the identification of high-risk antiphospholipid syndrome patients: a multicenter multiplatform study
.
J Thromb Haemost
.
2020
;
18
(
2
):
463
-
478
.
6.
Chayoua
W
,
Kelchtermans
H
,
Moore
GW
, et al
.
Detection of anti-cardiolipin and anti-beta2glycoprotein I antibodies differs between platforms without influence on association with clinical symptoms
.
Thromb Haemost
.
2019
;
119
(
5
):
797
-
806
.
7.
Devreese
KM
.
Antiphospholipid antibody testing and standardization
.
Int J Lab Hematol
.
2014
;
36
(
3
):
352
-
363
.
8.
Pelkmans
L
,
Kelchtermans
H
,
de Groot
PG
, et al
.
Variability in exposure of epitope G40-R43 of domain I in commercial anti-beta2-glycoprotein I IgG ELISAs
.
pLoS One
.
2013
;
8
:
e71402
.
9.
Favaloro
EJ
,
Wheatland
L
,
Jovanovich
S
,
Roberts-Thomson
P
,
Wong
RC
.
Internal quality control and external quality assurance in testing for antiphospholipid antibodies: part I —anticardiolipin and anti-beta2-glycoprotein I antibodies
.
Semin Thromb Hemost
.
2012
;
38
(
4
):
390
-
403
.
10.
de Laat-Kremers
RMW
,
Wahl
D
,
Zuily
S
, et al
.
Deciphered coagulation profile to diagnose the antiphospholipid syndrome using artificial intelligence
.
Thromb Res
.
2021
;
203
:
142
-
151
.
11.
Al Dieri
R
,
de Laat
B
,
Hemker
HC
.
Thrombin generation: what have we learned?
.
Blood Rev
.
2012
;
26
(
5
):
197
-
203
.
12.
Gehlen
R
,
Vandevelde
A
,
de Laat
B
,
Devreese
KMJ
.
Application of the thrombin generation assay in patients with antiphospholipid syndrome: a systematic review of the literature
.
Front Cardiovasc Med
.
2023
;
10
:
1075121
.
13.
Kremers
RMW
,
Zuily
S
,
Kelchtermans
H
, et al
.
Prothrombin conversion is accelerated in the antiphospholipid syndrome and insensitive to thrombomodulin
.
Blood Adv
.
2018
;
2
(
11
):
1315
-
1324
.
14.
Liestøl
S
,
Sandset
PM
,
Mowinckel
MC
,
Wisløff
F
.
Activated protein C resistance determined with a thrombin generation-based test is associated with thrombotic events in patients with lupus anticoagulants
.
J Thromb Haemost
.
2007
;
5
(
11
):
2204
-
2210
.
15.
Zuily
S
,
Ait Aissa
K
,
Membre
A
,
Regnault
V
,
Lecompte
T
,
Wahl
D
.
Thrombin generation in antiphospholipid syndrome
.
Lupus
.
2012
;
21
(
7
):
758
-
760
.
16.
Devreese
K
,
Peerlinck
K
,
Arnout
J
,
Hoylaerts
MF
.
Laboratory detection of the antiphospholipid syndrome via calibrated automated thrombography
.
Thromb Haemost
.
2009
;
101
(
1
):
185
-
196
.
17.
Devreese
K
,
Peerlinck
K
,
Hoylaerts
MF
.
Thrombotic risk assessment in the antiphospholipid syndrome requires more than the quantification of lupus anticoagulants
.
Blood
.
2010
;
115
(
4
):
870
-
878
.
18.
Kremers
RM
,
Peters
TC
,
Wagenvoord
RJ
,
Hemker
HC
.
The balance of pro- and anticoagulant processes underlying thrombin generation
.
J Thromb Haemost
.
2015
;
13
(
3
):
437
-
447
.
19.
Hassabis
D
,
Kumaran
D
,
Summerfield
C
,
Botvinick
M
.
Neuroscience-inspired artificial intelligence
.
Neuron
.
2017
;
95
(
2
):
245
-
258
.
20.
Rognoni
C
,
Lugli
M
,
Maleti
O
,
Tarricone
R
.
Clinical guidelines versus current clinical practice for the management of deep vein thrombosis
.
J Vasc Surg Venous Lymphat Disord
.
2021
;
9
(
5
):
1334
-
1344.e1
.
21.
Knight
JS
,
Branch
DW
,
Ortel
TL
.
Antiphospholipid syndrome: advances in diagnosis, pathogenesis, and management
.
BMJ
.
2023
;
380
:
e069717
.
22.
Tripodi
A
,
Cohen
H
,
Devreese
KMJ
.
Lupus anticoagulant detection in anticoagulated patients. Guidance from the Scientific and Standardization Committee for lupus anticoagulant/antiphospholipid antibodies of the International Society on Thrombosis and Haemostasis
.
J Thromb Haemost
.
2020
;
18
(
7
):
1569
-
1575
.
23.
Vandevelde
A
,
Devreese
KMJ
.
Laboratory diagnosis of antiphospholipid syndrome: insights and hindrances
.
J Clin Med
.
2022
;
11
(
8
):
2164
.
24.
Devreese
KMJ
,
de Groot
PG
,
de Laat
B
, et al
.
Guidance from the Scientific and Standardization Committee for lupus anticoagulant/antiphospholipid antibodies of the International Society on Thrombosis and Haemostasis: update of the guidelines for lupus anticoagulant detection and interpretation
.
J Thromb Haemost
.
2020
;
18
(
11
):
2828
-
2839
.
25.
Devreese
KM
,
Pierangeli
SS
,
de Laat
B
,
Tripodi
A
,
Atsumi
T
,
Ortel
TL
;
Subcommittee on Lupus Anticoagulant/Phospholipid/Dependent Antibodies
.
Testing for antiphospholipid antibodies with solid phase assays: guidance from the SSC of the ISTH
.
J Thromb Haemost
.
2014
;
12
(
5
):
792
-
795
.
26.
Rodziewicz
M
,
Cruz
DP
.
An update on the management of antiphospholipid syndrome
.
Ther Adv Musculoskelet Dis
.
2020
;
12
:
1759720x20910855
.
27.
Haas
S
,
Ten Cate
H
,
Accetta
G
, et al;
GARFIELD-AF Investigators
.
Quality of vitamin K antagonist control and 1-year outcomes in patients with atrial fibrillation: a global perspective from the GARFIELD-AF Registry
.
PLoS One
.
2016
;
11
(
10
):
e0164076
.
28.
Pengo
V
,
Tripodi
A
,
Reber
G
, et al;
Subcommittee on Lupus Anticoagulant/Antiphospholipid Antibody of the Scientific and Standardisation Committee of the International Society on Thrombosis and Haemostasis
.
Update of the guidelines for lupus anticoagulant detection. Subcommittee on Lupus Anticoagulant/Antiphospholipid Antibody of the Scientific and Standardisation Committee of the International Society on Thrombosis and Haemostasis
.
J Thromb Haemost
.
2009
;
7
(
10
):
1737
-
1740
.
29.
Tincani
A
,
Allegri
F
,
Sanmarco
M
, et al
.
Anticardiolipin antibody assay: a methodological analysis for a better consensus in routine determination—-a cooperative project of the European Antiphospholipid Forum
.
Thromb Haemost
.
2001
;
86
(
2
):
575
-
583
.
30.
De Moerloose
P
,
Reber
G
,
Musial
J
,
Arnout
J
.
Analytical and clinical performance of a new, automated assay panel for the diagnosis of antiphospholipid syndrome
.
J Thromb Haemost
.
2010
;
8
(
7
):
1540
-
1546
.
31.
Hemker
HC
,
Giesen
P
,
Al Dieri
R
, et al
.
Calibrated automated thrombin generation measurement in clotting plasma
.
Pathophysiol Haemost Thromb
.
2003
;
33
(
1
):
4
-
15
.
32.
Devreese
KMJ
,
Ortel
TL
,
Pengo
V
,
de Laat
B
;
Subcommittee on Lupus Anticoagulant/Antiphospholipid Antibodies
.
Laboratory criteria for antiphospholipid syndrome: communication from the SSC of the ISTH
.
J Thromb Haemost
.
2018
;
16
(
4
):
809
-
813
.
33.
Pengo
V
,
Denas
G
.
Diagnostics and treatment of thrombotic antiphospholipid syndrome (APS): a personal perspective
.
Thromb Res
.
2018
;
169
:
35
-
40
.
34.
Misra
DP
,
Ahmed
S
,
Goyal
M
,
Sharma
A
,
Agarwal
V
.
Venous thromboembolism in the inflammatory rheumatic diseases
.
Rheum Dis Clin North Am
.
2023
;
49
(
1
):
97
-
127
.
35.
Ferreira
LGR
,
Figueiredo
RC
,
das Graças Carvalho
M
,
Rios
DRA
.
Thrombin generation assay as a biomarker of cardiovascular outcomes and mortality: a narrative review
.
Thromb Res
.
2022
;
220
:
107
-
115
.
36.
van Hylckama Vlieg
A
,
Christiansen
SC
,
Luddington
R
,
Cannegieter
SC
,
Rosendaal
FR
,
Baglin
TP
.
Elevated endogenous thrombin potential is associated with an increased risk of a first deep venous thrombosis but not with the risk of recurrence
.
Br J Haematol
.
2007
;
138
(
6
):
769
-
774
.
37.
Wilson
WA
,
Gharavi
AE
,
Koike
T
, et al
.
International consensus statement on preliminary classification criteria for definite antiphospholipid syndrome: report of an international workshop
.
Arthritis Rheum
.
1999
;
42
(
7
):
1309
-
1311
.
38.
van Paridon
PCS
,
Panova-Noeva
M
,
van Oerle
R
, et al
.
Thrombin generation in cardiovascular disease and mortality—- results from the Gutenberg Health Study
.
Haematologica
.
2020
;
105
(
9
):
2327
-
2334
.
39.
Dielis
AW
,
Castoldi
E
,
Spronk
HM
, et al
.
Coagulation factors and the protein C system as determinants of thrombin generation in a normal population
.
J Thromb Haemost
.
2008
;
6
(
1
):
125
-
131
.
40.
Ninivaggi
M
,
de Laat-Kremers
R
,
Tripodi
A
, et al
.
Recommendations for the measurement of thrombin generation: communication from the ISTH SSC Subcommittee on Lupus Anticoagulant/Antiphospholipid Antibodies
.
J Thromb Haemost
.
2021
;
19
(
5
):
1372
-
1378
.

Author notes

Individual participant data will not be shared due to ethical restrictions.

The full-text version of this article contains a data supplement.

Supplemental data