Seeing Beneath the Rail: Sensing Technologies and Data Foundations for AI-Based Substructure Assessment

Executive Summary

Four sensing families now make the railroad track base measurable at scale: ground penetrating radar (GPR), which sees layer structure, fouling, and moisture; imaging systems paired with deep learning, which characterize ballast condition from visual evidence; vertical track deflection measurement, which infers the mechanical health of the support structure from its response to load; and satellite interferometric synthetic aperture radar (InSAR), which detects settlement across entire networks from orbit. Each modality answers a different question about the substructure, each has a distinct accuracy, coverage, and cost profile, and none is sufficient alone.

Machine learning is the common enabler across all four. It converts GPR returns into fouling estimates without expert manual interpretation, segments ballast imagery into quantitative condition measures, and supports the fusion of heterogeneous measurements into a single picture of substructure health. The constraint on this progress is data. Deep learning models are only as good as their training corpora, annotated substructure data remains scarce, and researchers have turned to synthetic data generation to fill the gap. The Federal Railroad Administration’s Automated Track Inspection Program (ATIP) archive, which the agency already makes available to support machine learning research, is the most practical foundation for a shared substructure data resource, but no annotation standard or governance structure yet exists to convert that archive into a training corpus for substructure models.

This paper details each modality, maps their complementarity, and closes with findings on the data foundations question. Its results feed the next installments directly: accuracy and validation evidence flows to Paper 2’s standards analysis, cost and coverage profiles flow to Paper 3’s economics, and data quality vulnerabilities flow to Paper 4’s hazard taxonomy.

1. The Measurement Problem

The foundational paper established what needs to be measured and why. The ballast layer holds ties in position, distributes wheel loads, and drains water; these functions degrade as the layer fouls with fine material, and fouled, poorly draining ballast accelerates track deterioration. The complete measurement problem for the track base spans four physical quantities: the fouling state of the ballast, the geometry of the substructure layers (ballast and subballast thickness and their variation along the track), the moisture regime (drainage performance and trapped water), and the mechanical response of the system (stiffness, deflection under load, and settlement over time).

No single instrument measures all four. The conventional direct method, excavating samples for sieve analysis, is subjective in site selection, labor intensive, and samples one location and depth that may not represent conditions along the track. The sensing families below each address part of the problem, and the engineering task, developed in Sections 2 through 4, is assembling them into a coherent measurement system.

2. Sensing Modalities

2.1 Ground Penetrating Radar

GPR transmits electromagnetic pulses into the trackbed and records the reflected and scattered energy returned from layer interfaces and material discontinuities. Two decades of research have established that properly applied GPR can observe ballast and subballast layer thickness, variations in thickness along the track, and water trapped in the ballast and in soft subgrade. The physics also defines the method’s core difficulties: the dielectric constant of ballast is not known a priori and varies with fouling and moisture, and the boundary between clean and fouled ballast is gradational rather than sharp, both of which limit the accuracy of conventional interpretation. Antenna selection matters as well; investigations of antenna quality and configuration have shown that the antenna largely determines inspection quality, and higher-frequency antennas exploit scattering from the void space between clean ballast particles, a signature that diminishes as voids fill with fines.

Machine learning addresses the interpretation burden directly. Research has demonstrated prediction of the ballast fouling index from GPR signal parameters developed in both the time and frequency domains, converting a signal-processing specialty into an automated estimation task. Condition scoring approaches have also been developed that integrate multiple GPR-derived variables, including dielectric permittivity, thickness scatter, boundary signal strength, and frequency spectrum area, into a single index that correlates with sieve-measured fouling, verified against field data. The combined picture is a modality that measures what no surface inspection can see, whose historical dependence on expert interpretation is being engineered away.

2.2 Imaging and Deep Learning Computer Vision

Where GPR infers internal condition from electromagnetics, imaging systems characterize ballast from visual evidence, and deep learning has transformed what that evidence yields. The research lineage begins with machine vision evaluation of ballast fouling conditions in the field, demonstrated under a Transportation Research Board Safety IDEA project, which established that image segmentation of ballast sections could classify degradation levels previously obtainable only through sieve analysis.

The current state of the art is the Ballast Scanning Vehicle developed with Federal Railroad Administration support: a platform employing three image acquisition devices to continuously capture high-quality scans of ballast cut sections, processed by a deep learning framework, enabling accurate and in-depth evaluation of continuous track sections rather than isolated sample points. Successive algorithm generations have moved from convolutional segmentation to vision transformer architectures for ballast condition classification. Related work applies deep learning to surface symptoms, including the detection and condition evaluation of mud spots, the visible expression of subsurface fouling and drainage failure. Machine vision is also already operational in the federal inspection fleet for track components: algorithms trained to detect infrastructure assets such as fasteners process imagery collected during ATIP surveys and identify possible defects.

The vision family’s defining strength is granularity: it produces particle-level and section-level condition evidence. Its defining dependency, developed in Section 4, is training data.

2.3 Vertical Track Deflection and Track Modulus

The third family measures the substructure’s mechanical response rather than its composition. Weak track support, including poor soil load-bearing capacity in the base and sub-base region, expresses itself as excessive vertical deflection under load, and FRA’s Track Research Program has pursued vertical track deflection measurement from moving vehicles as a complement to its instrumented geometry and gage restraint measurement fleet, with feasibility work confirming that the measurement can be made at an acceptable level of accuracy. The system lineage runs through FRA-sponsored university research on measuring vertical deflection from a moving railcar, and the methodology has been extended to use continuous deflection measurements from a moving loaded car to map subgrade condition along a railway line, connecting the measurement directly to the substructure layer this series addresses.

Deflection measurement occupies a distinctive position in the sensing portfolio: it is the only modality that measures the substructure’s function, load-bearing performance, rather than its state. A stiffness anomaly detected by deflection measurement is a functional finding that GPR and imaging can then explain compositionally.

2.4 Satellite InSAR

InSAR detects surface deformation by analyzing phase differences between radar images acquired on successive satellite passes, achieving millimeter-scale accuracy in deformation rate estimation across wide areas. Applied to railways, multi-temporal InSAR techniques, including persistent scatterer and small baseline subset methods, have monitored settlement along entire corridors, including high-speed lines crossing major subsidence regions. Critically for engineering credibility, InSAR-derived settlement measurements at railway transition zones have been cross-validated against measuring coach data and digital image correlation instruments, with the three techniques showing good correlation. Documented limitations bound the method: low-coherence areas such as vegetated embankments resist measurement, satellite revisit cycles limit temporal resolution, and validation against ground instruments remains necessary, with corner reflectors and GNSS stations used to strengthen results.

InSAR’s role in the portfolio is triage at network scale: it identifies where settlement is occurring across an entire system at low marginal cost, directing the higher-resolution, higher-cost modalities to the locations that warrant them.

2.5 Complementarity

The four families answer different questions, summarized below.

The complementarity is functional, not merely additive: InSAR locates, deflection confirms functional impairment, and GPR and imaging diagnose composition. That division of labor is the premise of the fusion work in Section 3.

3. Multi-Sensor Fusion

Recent research formalizes the integration. A 2024 study integrated GPR, InSAR, and machine learning into a combined track health monitoring approach, pairing GPR’s view of ballast fouling and moisture with InSAR’s rapid, network-wide diagnosis of settlement, and observing that integration of different technologies addresses the gaps that emerge from any single technology. European work had earlier proposed integrating persistent scatterer InSAR with GPR on operating railway segments, successfully identifying areas of subsidence for targeted investigation.

Fusion is where machine learning’s role expands from interpreting individual sensors to constructing the combined picture: co-registering measurements acquired at different times, resolutions, and geometries; learning the relationships between functional indicators (deflection, settlement) and compositional ones (fouling, moisture); and ultimately attributing observed track geometry degradation, the quantity regulation currently measures, to its subsurface causes. The foundational paper’s Layer 2 is built on exactly this attribution, and its feasibility depends on the data foundations examined next.

4. Data Foundations

4.1 The scarcity problem

Deep learning condition assessment is intrinsically data driven: the efficacy of these models is tied to the quality and comprehensiveness of annotated training datasets, and the ballast domain lacks the large public corpora that accelerated computer vision elsewhere. Annotating substructure data is expensive in a specific way: ground truth for a fouling model is a sieve analysis, ground truth for a GPR moisture model is an excavation, and ground truth for a settlement model is a survey campaign. Every label costs field work.

4.2 Synthetic augmentation

Researchers have responded by manufacturing data. A multi-dimensional ballast aggregate dataset has been constructed combining empirical field and laboratory imagery with synthetic data produced by a ballast particle generator, explicitly motivated by the need for comprehensive, meticulously annotated training data. Synthetic augmentation relieves scarcity but introduces its own question, whether models trained partly on generated particles transfer reliably to field conditions across the range of ballast materials, climates, and traffic profiles in service. That representativeness question is logged here and carried to Paper 4’s data hazard analysis.

4.3 The ATIP archive as a shared foundation

The most practical path to scale runs through data the government already collects. FRA’s ATIP fleet surveys over 150,000 miles of track annually, and the agency identifies machine learning and research among the uses its data supports, alongside safety trend analysis, state of good repair assessment, and risk analysis. The archive’s value for substructure work is that it is consistent, longitudinal, and independent: the same platforms, measuring the same parameters, over years. What it is not, yet, is a substructure training corpus. Converting it into one requires three things the current arrangement does not supply: an annotation standard linking measurement records to substructure ground truth where excavation or sampling has occurred; access terms that let railroads, researchers, and vendors train against common data without compromising security or commercial interests; and stewardship responsibility for label quality over time. Precedents for governed transportation data programs exist across the federal research enterprise, and defining the substructure-specific version is an institutional task, not a technical one.

5. Findings

Finding 1.1: The perception layer is modality-complete for the substructure measurement problem. Between GPR (composition and moisture), imaging (granular condition), deflection (mechanical function), and InSAR (network-scale settlement), every physical quantity identified in Section 1 has at least one demonstrated measurement path, and the modalities are complementary rather than redundant.

Finding 1.2: Machine learning has converted the two historically expert-limited modalities into automatable ones. GPR interpretation and ballast image assessment, each formerly dependent on specialist judgment, now have demonstrated learned pipelines, from fouling index prediction out of radar signal parameters to vision transformer condition classification of continuous ballast scans.

Finding 1.3: Field validation depth varies sharply by modality, and that variance will govern standardization. Deflection measurement carries FRA feasibility confirmation, InSAR carries cross-validation against measuring coach and image correlation instruments at transition zones, and GPR condition indices carry field verification in the published literature; deep learning ballast condition systems, the newest family, have the thinnest independent field validation record. Paper 2 takes this validation asymmetry as a starting condition.

Finding 1.4: Data, not algorithms, is the binding constraint, and the shared-corpus problem is institutional. Annotated substructure data is scarce because its ground truth requires field work, synthetic generation is a partial and unproven substitute at fleet scale, and the ATIP archive is the strongest candidate foundation for a shared corpus but lacks the annotation standard, access framework, and stewardship structure that would make it one.

6. Limitations

Quantitative accuracy comparisons across modalities could not be constructed from public sources on a common basis; published accuracy figures derive from differing test conditions, ground truth methods, and metrics, and any cross-modality accuracy table would have implied a comparability the underlying studies do not support. Commercial system performance claims were excluded as sole support throughout, consistent with the series’ sourcing standards. Fiber optic acoustic sensing and instrumented-particle methods, additional emerging substructure sensing approaches, were noted during research but deferred; they are assigned to the post-series research agenda rather than treated superficially here.

7. Threads Passed Forward

Three threads leave this paper for later installments. The validation asymmetry across modalities (Finding 1.3) becomes the central problem of Paper 2, which addresses how substructure measurements earn engineering and regulatory standing, including the question of how to validate measurements that have no human-inspection baseline. The coverage and cost characters summarized in Section 2.5 become inputs to Paper 3’s deployment economics, particularly the triage logic in which network-scale InSAR directs higher-cost modalities. The representativeness of synthetic training data, the label-quality stewardship problem, and coherence and drift limitations of the individual sensors (Sections 4.2 and 2.4) are carried to Paper 4 as entries in the data hazard class of its taxonomy.

Bibliography

Tier 1: Government and Oversight Sources

Federal Railroad Administration. ATIP: Automated Track Inspection Program (December 2025). TRB Quad Chart. Washington, DC: U.S. Department of Transportation, December 2025. https://railroads.dot.gov/sites/fra.dot.gov/files/2025-12/ATIP-TRB-QuadChart_Dec_2025.pdf.

Federal Railroad Administration. “Inspection Techniques.” Track and Structures Program Area. Washington, DC: U.S. Department of Transportation. https://railroads.dot.gov/program-areas/track-and-structures/inspection-techniques.

Federal Railroad Administration. Machine Vision Data Products for the Automated Track Inspection Program. Washington, DC: U.S. Department of Transportation, 2025. https://railroads.dot.gov/elibrary/machine-vision-data-products-automated-track-inspection-program.

Federal Railroad Administration. “Track Geometry Measurement System (TGMS) Inspections.” Notice of Proposed Rulemaking. Federal Register, October 24, 2024. https://www.federalregister.gov/documents/2024/10/24/2024-24153/track-geometry-measurement-system-tgms-inspections.

Farritor, Shane, and Mahmood Fateh. Measurement of Vertical Track Deflection from a Moving Rail Car. Washington, DC: U.S. Department of Transportation, Federal Railroad Administration, 2013.

Tier 2 and Tier 4: National Academies, University Transportation Research, and Peer-Reviewed Sources

“Advancing Railway Track Health Monitoring: Integrating GPR, InSAR and Machine Learning for Enhanced Asset Management.” Automation in Construction (2024). https://doi.org/10.1016/j.autcon.2024.105378.

Chang, Ling, et al. “Structural Health Monitoring of Railway Transition Zones Using Satellite Radar Data.” Sensors 18, no. 2 (2018): 413. https://doi.org/10.3390/s18020413.

“Development of Condition Assessment Index of Ballast Track Using Ground-Penetrating Radar (GPR).” Applied Sciences (2021). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8539047/.

Ding, Kelin, Jiayi Luo, Haohang Huang, John M. Hart, Issam I. A. Qamhia, and Erol Tutumluer. “Augmented Dataset for Vision-Based Analysis of Railroad Ballast via Multi-Dimensional Data Synthesis.” Algorithms 17, no. 8 (2024): 367. https://doi.org/10.3390/a17080367.

Fallah Nafari, Saeideh, Mustafa Gül, Michael T. Hendry, and J. J. Roger Cheng. “Estimation of Vertical Bending Stress in Rails Using Train-Mounted Vertical Track Deflection Measurement Systems.” Proceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit (2018). https://doi.org/10.1177/0954409717738444.

“Full Article: Ballast Fouling Inspection and Quantification with Ground Penetrating Radar (GPR).” International Journal of Rail Transportation (2022). https://doi.org/10.1080/23248378.2022.2064346.

Leng, Zhen, and Imad L. Al-Qadi. “Railroad Ballast Evaluation Using Ground-Penetrating Radar.” Transportation Research Record: Journal of the Transportation Research Board 2159 (2010): 110-117. https://doi.org/10.3141/2159-14.

Luo, Jiayi, Kelin Ding, Haohang Huang, John M. Hart, Issam I. A. Qamhia, Erol Tutumluer, Hugh Thompson, and Theodore R. Sussmann. “Toward Automated Field Ballast Condition Evaluation: Development of a Ballast Scanning Vehicle.” Transportation Research Record: Journal of the Transportation Research Board (2024). https://doi.org/10.1177/03611981231178302.

Luo, Jiayi, Haohang Huang, Kelin Ding, Issam I. A. Qamhia, Erol Tutumluer, John M. Hart, and Theodore R. Sussmann. “Toward Automated Field Ballast Condition Evaluation: Algorithm Development Using a Vision Transformer Framework.” Transportation Research Record: Journal of the Transportation Research Board (2023). https://doi.org/10.1177/03611981231161350.

“Condition Evaluation of Mud Spots in Railroad Ballast Using Deep Learning.” Journal of Infrastructure Systems 31, no. 4 (2025). https://doi.org/10.1061/JITSE4.ISENG-2700.

“Monitoring Differential Subsidence along the Beijing-Tianjin Intercity Railway with Multiband SAR Data.” Remote Sensing (2019). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6888550/.

Roghani, Alireza, and Michael T. Hendry. “Continuous Vertical Track Deflection Measurements to Map Subgrade Condition along a Railway Line: Methodology and Case Studies.” Journal of Transportation Engineering 142, no. 12 (2016). https://doi.org/10.1061/(ASCE)TE.1943-5436.0000892.

Selig, Ernest T., and John M. Waters. Track Geotechnology and Substructure Management. London: Thomas Telford, 1994.

Tutumluer, Erol, Narendra Ahuja, John M. Hart, Maziar Moaveni, Haohang Huang, Zixu Zhao, and Sagar Shah. Field Evaluation of Ballast Fouling Conditions Using Machine Vision. Safety IDEA Project 27 Final Report. Washington, DC: Transportation Research Board of the National Academies, 2017.