Performing inference on new data
Refer to the tutorials (Linux, Windows) for installation instructions.
To run inference, you will need:
A checkpoint of a trained lightning module implementing model logic (class
simlearner3d.models.generic_model.Model)A pair of epipolar images
An installed and working version of our micmac repository.
Transforming .ckpt model to scripted .pt models
Our our micmac version leverages learned models together with PyTorch c++ api to perform large scale stereo image matching. In order to use trained models, you need to script the model checkpoint from pytorchlightning to a .pt model. This is performed using the following command line:
python run.py \
task.task_name=extract_pt \
model.ckpt_path={/path/to/checkpoint.ckpt}
Run inference on a pair of epipolar images
Inference routines are implemented in C++ into the open photogrammetry software MicMac.
It tackles large scale stereo matching for large frame aerial imagery and EHR Satellite stereo reconstruction. Our models are embedded into a hierachical multi-resolution pipeline where lower resolution surface reconstructions serve as a prior for the next level reconstruction. This strategy is robust and allows integrating hand-crafted similarity metrics like (ncc, census,…) for lower resolution low context matching.
To perform hierarchical matching using our similarity learning models, you need to:
git clone our repository and install
micmacandMicMac V2after downloading plateform specific torch cpp libraries. You may need to modifiyTORCH_INSTALL_PREFIXin CMakeLists.txt to point tolibtorchfolder. For our installation, we set it to${MMVII_SOURCE_DIR}/libtorchcompile MMVII and micmac with libtorch
MicMac uses
xml specificationsfiles for hierachical dense image matching, we provide an example xml configuration file:
<ParamMICMAC>
<DicoLoc>
<Symb> ZReg=0.005 </Symb>
<Symb> DefCor=0.4 </Symb>
<Symb> CostTrans=1.0 </Symb>
<Symb> SzW=3 </Symb>
<Symb> PenteMax=3.0 </Symb>
<Symb> Interpol=eInterpolBiLin </Symb>
<Symb> DoEpi=true </Symb>
<Symb> Ori=Epi </Symb>
<Symb> Im=TEST </Symb>
<Symb> Im1=Im1DUBL.tif </Symb>
<Symb> Im2=Im2DUBL.tif </Symb>
<Symb> Modele=XXX</Symb>
<Symb> WITH_MODELE=true </Symb>
<Symb> IncPix=100 </Symb>
<Symb> NbDirProg=7 </Symb>
<Symb> ExtImIn=tif </Symb>
<Symb> Purge=true </Symb>
<Symb> NbProc=4 </Symb>
<Symb> OnCuda=false</Symb>
<Symb> UseMLP=false</Symb>
<!-- Parametres calcule -->
<Symb> DirMEC=${Modele} </Symb>
<Symb> DirPyram=Pyram/ </Symb>
</DicoLoc>
<Section_Terrain>
<IntervParalaxe>
<Px1IncCalc> ${IncPix} </Px1IncCalc>
</IntervParalaxe>
</Section_Terrain>
<Section_PriseDeVue>
<GeomImages> eGeomImage_EpipolairePure </GeomImages>
<Images>
<Im1> ${Im1} </Im1>
<Im2> ${Im2} </Im2>
</Images>
<MasqImageIn>
<OneMasqueImage>
<PatternSel> (.*)\.${ExtImIn} </PatternSel>
<NomMasq> $1_Masq.tif </NomMasq>
</OneMasqueImage>
<AcceptNonExistingFile> true </AcceptNonExistingFile>
</MasqImageIn>
</Section_PriseDeVue>
<!-- *************************************************************
Parametres fixant le comportement
de l'algo de mise en correspondance
-->
<Section_MEC>
<ChantierFullImage1> true </ChantierFullImage1>
<EtapeMEC>
<DeZoom> -1 </DeZoom>
<ModeInterpolation> ${Interpol} </ModeInterpolation>
<!-- param correl -->
<SzW> ${SzW} </SzW>
<AlgoRegul> eAlgo2PrgDyn </AlgoRegul>
<Px1Pas> 1 </Px1Pas>
<Px1DilatAlti> 7 </Px1DilatAlti>
<Px1DilatPlani> 3 </Px1DilatPlani>
<Px1Regul> ${ZReg} </Px1Regul>
<GenImagesCorrel> true </GenImagesCorrel>
<SsResolOptim> 1 </SsResolOptim>
<CoxRoyUChar> false </CoxRoyUChar>
<ModulationProgDyn>
<EtapeProgDyn>
<ModeAgreg> ePrgDAgrSomme </ModeAgreg>
<NbDir> ${NbDirProg} </NbDir>
</EtapeProgDyn>
<Px1PenteMax> ${PenteMax} </Px1PenteMax>
<ArgMaskAuto>
<ValDefCorrel> ${DefCor} </ValDefCorrel>
<CostTrans> ${CostTrans} </CostTrans>
<ReInjectMask> false </ReInjectMask>
</ArgMaskAuto>
</ModulationProgDyn>
</EtapeMEC>
<!-- <EtapeMEC> <DeZoom> 16 </DeZoom> </EtapeMEC> -->
<!-- <EtapeMEC> <DeZoom> 64 </DeZoom> </EtapeMEC>
<EtapeMEC> <DeZoom> 32 </DeZoom> </EtapeMEC> -->
<EtapeMEC> <DeZoom> 16 </DeZoom> </EtapeMEC>
<EtapeMEC> <DeZoom> 8 </DeZoom>
<CorrelAdHoc>
<SzBlocAH> 40000000 </SzBlocAH>
<TypeCAH>
<ScoreLearnedMMVII >
<FileModeleCost> MVCNNCorrel</FileModeleCost>
<FileModeleParams>../MODEL_MSAFF_AERIAL_DECISION/.*.pt</FileModeleParams>
<FileModeleArch>UnetMLPMatcher</FileModeleArch>
<Cuda> ${OnCuda} </Cuda>
<UsePredicNet> ${UseMLP} </UsePredicNet>
</ScoreLearnedMMVII>
</TypeCAH>
</CorrelAdHoc>
</EtapeMEC>
<EtapeMEC> <DeZoom> 4 </DeZoom>
<CorrelAdHoc>
<SzBlocAH> 40000000 </SzBlocAH>
<TypeCAH>
<ScoreLearnedMMVII >
<FileModeleCost> MVCNNCorrel</FileModeleCost>
<FileModeleParams>../MODEL_MSAFF_AERIAL_DECISION/.*.pt</FileModeleParams>
<FileModeleArch>UnetMLPMatcher</FileModeleArch>
<Cuda> ${OnCuda} </Cuda>
<UsePredicNet> ${UseMLP} </UsePredicNet>
</ScoreLearnedMMVII>
</TypeCAH>
</CorrelAdHoc>
</EtapeMEC>
<EtapeMEC>
<DeZoom> 2 </DeZoom>
<CorrelAdHoc>
<SzBlocAH> 40000000 </SzBlocAH>
<TypeCAH>
<ScoreLearnedMMVII >
<FileModeleCost> MVCNNCorrel</FileModeleCost>
<FileModeleParams>../MODEL_MSAFF_AERIAL_DECISION/.*.pt</FileModeleParams>
<FileModeleArch>UnetMLPMatcher</FileModeleArch>
<Cuda> ${OnCuda} </Cuda>
<UsePredicNet> ${UseMLP} </UsePredicNet>
</ScoreLearnedMMVII>
</TypeCAH>
</CorrelAdHoc>
</EtapeMEC>
<EtapeMEC>
<DeZoom> 1 </DeZoom>
<CorrelAdHoc>
<SzBlocAH> 40000000 </SzBlocAH>
<TypeCAH>
<ScoreLearnedMMVII >
<FileModeleCost> MVCNNCorrel</FileModeleCost>
<FileModeleParams>../MODEL_MSAFF_AERIAL_DECISION/.*.pt</FileModeleParams>
<FileModeleArch>UnetMLPMatcher</FileModeleArch>
<Cuda> ${OnCuda} </Cuda>
<UsePredicNet> ${UseMLP} </UsePredicNet>
</ScoreLearnedMMVII>
</TypeCAH>
</CorrelAdHoc>
</EtapeMEC>
<!-- <EtapeMEC>
<DeZoom > 1 </DeZoom>
<Px1Pas> 0.5 </Px1Pas>
</EtapeMEC> -->
<EtapeMEC>
<DeZoom> 1 </DeZoom>
<Px1Pas> 1.0 </Px1Pas>
<AlgoRegul> eAlgoDequant </AlgoRegul>
</EtapeMEC>
<HighPrecPyrIm> false </HighPrecPyrIm>
<TypePyramImage>
<Resol > 1 </Resol>
<DivIm> 1 </DivIm>
<TypeEl> eFloat32Bits </TypeEl>
</TypePyramImage>
</Section_MEC>
<!-- *************************************************************
Parametres fixant les resultats
devant etre produits par l'algo
-->
<Section_Results>
<GeomMNT> eGeomPxBiDim </GeomMNT>
<ZoomMakeTA> 16 </ZoomMakeTA>
<GammaVisu> 2.0 </GammaVisu>
<ZoomVisuLiaison> -1 </ZoomVisuLiaison>
</Section_Results>
<!-- *************************************************************
Parametres lies a la gestions
du "chantier" sur la machine
-->
<Section_WorkSpace>
<WorkDir> ThisDir </WorkDir>
<TmpMEC> ${DirMEC}/ </TmpMEC>
<TmpResult> ${DirMEC}/ </TmpResult>
<TmpPyr> ${DirPyram} </TmpPyr>
<PurgeMECResultBefore> ${Purge} </PurgeMECResultBefore>
<ByProcess> ${NbProc} </ByProcess>
<AvalaibleMemory> 2048 </AvalaibleMemory>
<SzDalleMin> 540 </SzDalleMin>
<SzDalleMax> 512 </SzDalleMax>
<SzRecouvrtDalles> 20 </SzRecouvrtDalles>
<SzMinDecomposCalc> 40 </SzMinDecomposCalc>
<ComprMasque> eComprTiff_None </ComprMasque>
</Section_WorkSpace>
<Section_Vrac>
<DebugMM> true</DebugMM>
</Section_Vrac>
</ParamMICMAC>
The aformentionned xml file activates our models from
Zoom 4, i.e models are in charge of reconstruction for zooms: 4,2,1. This is accomplished with and adhoc correlation that calls a set of .pt models (feature extraction + mlp).
<CorrelAdHoc>
<SzBlocAH> 40000000 </SzBlocAH>
<TypeCAH>
<ScoreLearnedMMVII >
<FileModeleCost> MVCNNCorrel</FileModeleCost>
<FileModeleParams>../MODEL_MSAFF_AERIAL_DECISION/.*.pt</FileModeleParams>
<FileModeleArch>UnetMLPMatcher</FileModeleArch>
<Cuda> ${OnCuda} </Cuda>
<UsePredicNet> ${UseMLP} </UsePredicNet>
</ScoreLearnedMMVII>
</TypeCAH>
</CorrelAdHoc>
Parametrization: To launch stereo matching between a pair of epipolar images,
mm3d MICMAC File.xml +Im1=Im1.tif +Im2=Im2.tif +DirMEC=MEC/ +OnCuda=0 +UsePredicNet=1