公开(公告)号：US08675935B2

公开(公告)日：2014-03-18

申请号：US13297967

申请日：2011-11-16

Applicant: William E. Higgins ,  Scott A. Merritt ,  Lav Rai

Inventor： William E. Higgins ,  Scott A. Merritt ,  Lav Rai

IPC: G06K9/36

CPC classification number: A61B1/00147 ,  A61B1/2676 ,  A61B5/415 ,  G06K9/00208 ,  G06K9/00214 ,  G06K9/6206 ,  G06T3/0081 ,  G06T7/33 ,  G06T2207/10028 ,  G06T2207/10068 ,  G06T2207/10081 ,  G06T2207/30061

Abstract: Fast and continuous registration between two imaging modalities makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize video cameras and register the two sources. A set of reference images are computed or captured within a known environment, with corresponding depth maps and image gradients defining a reference source. Given one frame from a real-time or near-real time video feed, and starting from an initial guess of viewpoint, a real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. Steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source.

Abstract translation: 两种成像模式之间的快速和连续的配准使得可以完全确定实时或接近实时帧率的多个源之间的刚性变换，以便本地化摄像机并注册两个源。 在已知环境中计算或捕获一组参考图像，其中相应的深度图和图像梯度定义参考源。 给定来自实时或近实时视频馈送的一帧，并且从视点的初始猜测开始，将实时视频帧扭曲到参考源的最近观看站点。 在翘曲的视频帧和参考图像之间计算图像差异。 对于每个帧重复步骤，直到视点收敛或下一个视频帧变得可用。 最后的观点给出了在该特定视频帧处的摄像机与参考源之间的相对旋转和平移的估计。

公开(公告)号：US20110184238A1

公开(公告)日：2011-07-28

申请号：US13016093

申请日：2011-01-28

Applicant: William E. Higgins ,  Rahul Khare ,  Scott A. Merritt

Inventor： William E. Higgins ,  Rahul Khare ,  Scott A. Merritt

IPC: A61B1/00

CPC classification number: A61B1/2676 ,  A61B1/00009 ,  G06K9/00208 ,  G06K9/00214 ,  G06K9/6206 ,  G06T7/75 ,  G06T2207/10068 ,  G06T2207/30061

Abstract: A global registration system and method identifies bronchoscope position without the need for significant bronchoscope maneuvers, technician intervention, or electromagnetic sensors. Virtual bronchoscopy (VB) renderings of a 3D airway tree are obtained including VB views of branch positions within the airway tree. At least one real bronchoscopic (RB) video frame is received from a bronchoscope inserted into the airway tree. An algorithm according to the invention is executed on a computer to identify the several most likely branch positions having a VB view closest to the received RB view, and the 3D position of the bronchoscope within the airway tree is determined in accordance with the branch position identified in the VB view. The preferred embodiment involves a fast local registration search over all the branches in a global airway-bifurcation search space, with the weighted normalized sum of squares distance metric used for finding the best match.

Abstract translation: 全球注册系统和方法识别支气管镜位置，而不需要显着的支气管镜手术，技术人员干预或电磁传感器。 获得3D气道树的虚拟支气管镜（VB）效果图，包括气道树内分支位置的VB视图。 从插入气道树的支气管镜接收至少一个真正的支气管镜（RB）视频帧。 根据本发明的算法在计算机上执行以识别具有最接近所接收的RB视图的VB视图的几个最可能的分支位置，并且根据所识别的分支位置来确定气道树内的支气管镜的3D位置 在VB视图中。 优选实施例涉及在全球气道分岔搜索空间中的所有分支上的快速本地登记搜索，其中用于找到最佳匹配的加权归一化平方和距离度量。

公开(公告)号：US20110128352A1

公开(公告)日：2011-06-02

申请号：US13022241

申请日：2011-02-07

Applicant: William E. Higgins ,  Scott A. Merritt ,  Lav Rai

Inventor： William E. Higgins ,  Scott A. Merritt ,  Lav Rai

IPC: A61B1/04 ,  H04N13/02 ,  G06K9/00

CPC classification number: A61B1/00147 ,  A61B1/2676 ,  A61B5/415 ,  G06K9/00208 ,  G06K9/00214 ,  G06K9/6206 ,  G06T3/0081 ,  G06T7/33 ,  G06T2207/10028 ,  G06T2207/10068 ,  G06T2207/10081 ,  G06T2207/30061

Abstract: A novel framework for fast and continuous registration between two imaging modalities is disclosed. The approach makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize the cameras and register the two sources. A disclosed example includes computing or capturing a set of reference images within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source. The second source is a real-time or near-real time source which may include a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. The viewpoint is updated via a Gauss-Newton parameter update and certain of the steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source. The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.

Abstract translation: 公开了一种用于两种成像模式之间快速连续配准的新颖框架。 该方法使得可以完全确定实时或接近实时帧速率的多个源之间的刚性变换，以便本地化摄像机并注册两个源。 所公开的示例包括在已知环境内计算或捕获一组参考图像，其具有相应的深度图和图像梯度。 这些图像和深度图的集合构成参考源。 第二个来源是实时或近实时源，其中可能包含实况视频馈送。 从这个视频馈送提供一帧，并从最初的观点猜测开始，实时视频帧被扭曲到参考源的最近的观看站点。 在翘曲的视频帧和参考图像之间计算图像差异。 通过高斯 - 牛顿参数更新来更新视点，并且对于每个帧重复某些步骤，直到视点收敛或下一个视频帧变得可用。 最后的观点给出了在该特定视频帧处的摄像机与参考源之间的相对旋转和平移的估计。 本发明具有深远的应用，特别是在辅助内窥镜检查领域，包括支气管镜检查和结肠镜检查。 其他应用包括空中和地面导航。

公开(公告)号：US06404793B1

公开(公告)日：2002-06-11

申请号：US09650675

申请日：2000-08-30

Applicant: Scott A. Merritt

Inventor： Scott A. Merritt

IPC: H01S500

CPC classification number: H01S5/026 ,  G02F1/295 ,  G02F2201/04 ,  G02F2201/05 ,  H01S5/06243 ,  H01S5/1032 ,  H01S5/1064 ,  H01S5/14

Abstract: A novel device and method of beam steering for semiconductor lasers or optical amplifiers is disclosed. The method of the present invention achieves high signal extinction ratios, high speed, low chirp modulation by biasing a multi-lateral mode beam steering section. The device of the present invention comprises an active single vertical and lateral mode optical waveguide, a multi-lateral mode waveguide, and a mode converter. The mode converter efficiently couples output from an active single mode waveguide to two or more modes of a multi-lateral mode waveguide. Two guided modes arrive at a device facet with a particular intermodal phase difference based on initial mode phasing, multi-lateral mode waveguide length and modal dispersion properties, and facet angle. Beam steering is achieved through carrier antiguiding effect by injecting current into the multi-lateral mode waveguide from the mode converter thus changing the intermodal dispersion. Changing the intermodal phase difference changes the direction of beam propagation relative to the device facet, providing enhanced beam steering.

Abstract translation: 公开了一种用于半导体激光器或光放大器的光束转向的新型装置和方法。 本发明的方法通过偏置多侧模式光束转向部分实现了高信号消光比，高速度，低啁啾调制。 本发明的器件包括有源单个垂直和横向模式光波导，多侧模式波导和模式转换器。 模式转换器将来自有源单模波导的输出有效地耦合到多侧波导的两个或多个模式。 基于初始模式定相，多波长波导长度和模态色散特性以及小平面角度，两个引导模式到达具有特定联合相位差的器件面。 通过从模式转换器将电流注入到多侧模式波导中，通过载波防护效应实现波束转向，从而改变多模式色散。 改变联运相位差会相对于装置小面改变光束传播的方向，从而提供增强的光束转向。

公开(公告)号：US06169757A

公开(公告)日：2001-01-02

申请号：US08938368

申请日：1997-09-26

Applicant: Scott A. Merritt

Inventor： Scott A. Merritt

IPC: H01S500

CPC classification number: H01S5/026 ,  G02F1/295 ,  G02F2201/04 ,  G02F2201/05 ,  H01S5/06243 ,  H01S5/1032 ,  H01S5/1064 ,  H01S5/14

Abstract: A novel device and method of beam steering for semiconductor lasers or optical amplifiers is disclosed. The method of the present invention achieves high signal extinction ratios, high speed, low chirp modulation by biasing a multi-lateral mode beam steering section. The device of the present invention comprises an active single vertical and lateral mode optical waveguide, a multi-lateral mode waveguide, and a mode converter. The mode converter efficiently couples output from an active single mode waveguide to two or more modes of a multi-lateral mode waveguide. Two guided modes arrive at a device facet with a particular intermodal phase difference based on initial mode phasing, multi-lateral mode waveguide length and modal dispersion properties, and facet angle. Beam steering is achieved through carrier antiguiding effect by injecting current into the multi-lateral mode waveguide from the mode converter thus changing the intermodal dispersion. Changing the intermodal phase difference changes the direction of beam propagation relative to the device facet, providing enhanced beam steering.

Abstract translation: 公开了一种用于半导体激光器或光放大器的光束转向的新型装置和方法。 本发明的方法通过偏置多侧模式光束转向部分实现了高信号消光比，高速度，低啁啾调制。 本发明的器件包括有源单个垂直和横向模式光波导，多侧模式波导和模式转换器。 模式转换器将来自有源单模波导的输出有效地耦合到多侧波导的两个或多个模式。 基于初始模式定相，多波长波导长度和模态色散特性以及小平面角度，两个引导模式到达具有特定联合相位差的器件面。 通过从模式转换器将电流注入到多侧模式波导中，通过载波防护效应实现波束转向，从而改变多模式色散。 改变联运相位差会相对于装置小面改变光束传播的方向，从而提供增强的光束转向。

公开(公告)号：US08064669B2

公开(公告)日：2011-11-22

申请号：US13022241

申请日：2011-02-07

Applicant: William E. Higgins ,  Scott A. Merritt ,  Lav Rai

Inventor： William E. Higgins ,  Scott A. Merritt ,  Lav Rai

IPC: G06K9/00 ,  G06K9/32 ,  A61B5/05

CPC classification number: A61B1/00147 ,  A61B1/2676 ,  A61B5/415 ,  G06K9/00208 ,  G06K9/00214 ,  G06K9/6206 ,  G06T3/0081 ,  G06T7/33 ,  G06T2207/10028 ,  G06T2207/10068 ,  G06T2207/10081 ,  G06T2207/30061

Abstract: A novel framework for fast and continuous registration between two imaging modalities is disclosed. The approach makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize the cameras and register the two sources. A disclosed example includes computing or capturing a set of reference images within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source. The second source is a real-time or near-real time source which may include a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. The viewpoint is updated via a Gauss-Newton parameter update and certain of the steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source. The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.

Abstract translation: 公开了一种用于两种成像模式之间快速连续配准的新颖框架。 该方法使得可以完全确定实时或接近实时帧速率的多个源之间的刚性变换，以便本地化摄像机并注册两个源。 所公开的示例包括在已知环境内计算或捕获一组参考图像，其具有相应的深度图和图像梯度。 这些图像和深度图的集合构成参考源。 第二个来源是实时或近实时源，其中可能包含实况视频馈送。 从这个视频馈送提供一帧，并从最初的观点猜测开始，实时视频帧被扭曲到参考源的最近的观看站点。 在翘曲的视频帧和参考图像之间计算图像差异。 通过高斯 - 牛顿参数更新来更新视点，并且对于每个帧重复某些步骤，直到视点收敛或下一个视频帧变得可用。 最后的观点给出了在该特定视频帧处的摄像机与参考源之间的相对旋转和平移的估计。 本发明具有深远的应用，特别是在辅助内窥镜检查领域，包括支气管镜检查和结肠镜检查。 其他应用包括空中和地面导航。

公开(公告)号：US07889905B2

公开(公告)日：2011-02-15

申请号：US11437230

申请日：2006-05-19

Applicant: William E. Higgins ,  Scott A. Merritt ,  Lav Rai

Inventor： William E. Higgins ,  Scott A. Merritt ,  Lav Rai

IPC: G06K9/00 ,  G06K9/32 ,  A61B5/05

CPC classification number: A61B1/00147 ,  A61B1/2676 ,  A61B5/415 ,  G06K9/00208 ,  G06K9/00214 ,  G06K9/6206 ,  G06T3/0081 ,  G06T7/33 ,  G06T2207/10028 ,  G06T2207/10068 ,  G06T2207/10081 ,  G06T2207/30061

Abstract: A novel framework for fast and continuous registration between two imaging modalities is disclosed. The approach makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize the cameras and register the two sources. A disclosed example includes computing or capturing a set of reference images within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source. The second source is a real-time or near-real time source which may include a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. The viewpoint is updated via a Gauss-Newton parameter update and certain of the steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source. The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.

Abstract translation: 公开了一种用于两种成像模式之间快速连续配准的新颖框架。 该方法使得可以完全确定实时或接近实时帧速率的多个源之间的刚性变换，以便本地化摄像机并注册两个源。 所公开的示例包括在已知环境内计算或捕获一组参考图像，其具有相应的深度图和图像梯度。 这些图像和深度图的集合构成参考源。 第二个来源是实时或近实时源，其中可能包含实况视频馈送。 从这个视频馈送提供一帧，并从最初的观点猜测开始，实时视频帧被扭曲到参考源的最近的观看站点。 在翘曲的视频帧和参考图像之间计算图像差异。 通过高斯 - 牛顿参数更新来更新视点，并且对于每个帧重复某些步骤，直到视点收敛或下一个视频帧变得可用。 最后的观点给出了在该特定视频帧处的摄像机与参考源之间的相对旋转和平移的估计。 本发明具有深远的应用，特别是在辅助内窥镜检查领域，包括支气管镜检查和结肠镜检查。 其他应用包括空中和地面导航。

公开(公告)号：US20080207997A1

公开(公告)日：2008-08-28

申请号：US12022620

申请日：2008-01-30

Applicant: William E. Higgins ,  Scott A. Merritt ,  Lav Rai ,  Jason D. Gibbs ,  Kun-Chang Yu

Inventor： William E. Higgins ,  Scott A. Merritt ,  Lav Rai ,  Jason D. Gibbs ,  Kun-Chang Yu

IPC: A61B1/01

CPC classification number: A61B1/0005 ,  A61B1/00009 ,  A61B34/20 ,  A61B34/25 ,  A61B90/36 ,  A61B90/361 ,  A61B2034/105 ,  A61B2034/107 ,  A61B2090/365

Abstract: Methods and apparatus provide continuous guidance of endoscopy during a live procedure. A data-set based on 3D image data is pre-computed including reference information representative of a predefined route through a body organ to a final destination. A plurality of live real endoscopic (RE) images are displayed as an operator maneuvers an endoscope within the body organ. A registration and tracking algorithm registers the data-set to one or more of the RE images and continuously maintains the registration as the endoscope is locally maneuvered. Additional information related to the final destination is then presented enabling the endoscope operator to decide on a final maneuver for the procedure. The reference information may include 3D organ surfaces, 3D routes through an organ system, or 3D regions of interest (ROIs), as well as a virtual endoscopic (VE) image generated from the precomputed data-set. The preferred method includes the step of superimposing one or both of the 3D routes and ROIs on one or both of the RE and VE images. The 3D organ surfaces and routes may correspond to the surfaces and paths of a tracheobronchial airway tree extracted, for example, from 3D MDCT images of the chest.

Abstract translation: 方法和装置在实时程序中提供内窥镜检查的连续指导。 基于3D图像数据的数据集被预先计算，包括表示通过身体器官到最终目的地的预定义路线的参考信息。 多个实时内窥镜（RE）图像作为操作者显示在身体器官内的内窥镜。 注册和跟踪算法将数据集注册到一个或多个RE图像，并且在内窥镜被局部操纵时连续地保持注册。 然后呈现与最终目的地相关的附加信息，使得内窥镜操作者能够决定该程序的最终操纵。 参考信息可以包括3D器官表面，通过器官系统的3D路线或感兴趣的3D区域（ROI）以及从预先计算的数据集生成的虚拟内窥镜（VE）图像。 优选的方法包括将一个或两个3D路线和ROI叠加在RE和VE图像中的一个或两个上的步骤。 3D器官表面和路线可以对应于例如从胸部的3D MDCT图像提取的气管支气管气道树的表面和路径。

公开(公告)号：US20120082351A1

公开(公告)日：2012-04-05

申请号：US13297967

申请日：2011-11-16

Applicant: William E. Higgins ,  Scott A. Merritt ,  Lav Rai

Inventor： William E. Higgins ,  Scott A. Merritt ,  Lav Rai

IPC: G06K9/00

CPC classification number: A61B1/00147 ,  A61B1/2676 ,  A61B5/415 ,  G06K9/00208 ,  G06K9/00214 ,  G06K9/6206 ,  G06T3/0081 ,  G06T7/33 ,  G06T2207/10028 ,  G06T2207/10068 ,  G06T2207/10081 ,  G06T2207/30061

Abstract: Fast and continuous registration between two imaging modalities makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize video cameras and register the two sources. A set of reference images are computed or captured within a known environment, with corresponding depth maps and image gradients defining a reference source. Given one frame from a real-time or near-real time video feed, and starting from an initial guess of viewpoint, a real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. Steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source.

Abstract translation: 两种成像模式之间的快速和连续的配准使得可以完全确定实时或接近实时帧率的多个源之间的刚性变换，以便本地化摄像机并注册两个源。 在已知环境中计算或捕获一组参考图像，其中相应的深度图和图像梯度定义参考源。 给定来自实时或近实时视频馈送的一帧，并且从视点的初始猜测开始，将实时视频帧扭曲到参考源的最近观看站点。 在翘曲的视频帧和参考图像之间计算图像差异。 对于每个帧重复步骤，直到视点收敛或下一个视频帧变得可用。 最后的观点给出了在该特定视频帧处的摄像机与参考源之间的相对旋转和平移的估计。

公开(公告)号：US07756563B2

公开(公告)日：2010-07-13

申请号：US11437229

申请日：2006-05-19

Applicant: William E. Higgins ,  Scott A. Merritt ,  Lav Rai

Inventor： William E. Higgins ,  Scott A. Merritt ,  Lav Rai

IPC: A61B5/05

CPC classification number: A61B5/06 ,  A61B1/00009 ,  A61B5/415 ,  A61B5/418 ,  A61B6/032 ,  A61B6/466 ,  A61B6/50 ,  A61B34/10 ,  A61B34/20 ,  A61B90/36 ,  A61B90/361 ,  A61B2034/105 ,  A61B2034/2065 ,  A61B2090/364 ,  A61B2090/3762 ,  G06K9/00208 ,  G06K9/00214 ,  G06K9/6206

Abstract: A method provides guidance to the physician during a live bronchoscopy or other endoscopic procedures. The 3D motion of the bronchoscope is estimated using a fast coarse tracking step followed by a fine registration step. The tracking is based on finding a set of corresponding feature points across a plurality of consecutive bronchoscopic video frames, then estimating for the new pose of the bronchoscope. In the preferred embodiment the pose estimation is based on linearization of the rotation matrix. By giving a set of corresponding points across the current bronchoscopic video image, and the CT-based virtual image as an input, the same method can also be used for manual registration. The fine registration step is preferably a gradient-based Gauss-Newton method that maximizes the correlation between the bronchoscopic video image and the CT-based virtual image. The continuous guidance is provided by estimating the 3D motion of the bronchoscope in a loop. Since depth-map information is available, tracking can be done by solving a 3D-2D pose estimation problem. A 3D-2D pose estimation problem is more constrained than a 2D-2D pose estimation problem and does not suffer from the limitations associated with computing an essential matrix. The use of correlation-based cost, instead of mutual information as a registration cost, makes it simpler to use gradient-based methods for registration.

Abstract translation: 一种方法在活支气管镜检查或其他内窥镜手术过程中向医生提供指导。 支气管镜的3D运动使用快速粗跟踪步骤，接着是精细配准步骤来估计。 跟踪是基于在多个连续支气管镜视频帧之间找到一组对应的特征点，然后估计支气管镜的新姿势。 在优选实施例中，姿态估计基于旋转矩阵的线性化。 通过在目前的支气管镜视频图像上提供一组对应的点，并且基于CT的虚像作为输入，同样的方法也可以用于手动注册。 精细登记步骤优选地是使支气管镜视频图像和基于CT的虚拟图像之间的相关性最大化的基于梯度的高斯 - 牛顿方法。 通过在循环中估计支气管镜的3D运动来提供连续的指导。 由于深度图信息可用，可以通过解决3D-2D姿态估计问题来进行跟踪。 3D-2D姿态估计问题比2D-2D姿态估计问题更受约束，并且不受与计算基本矩阵相关联的限制的影响。 使用基于相关性的成本，而不是相互信息作为注册成本，使得使用基于梯度的方法进行注册变得更简单。