Interview With General Motors Lead Engineer For Sensor Fusion Jim Nickolaou

Posted Aug 26, 2012

Earlier this month General Motors conducted an experiment on the Brooklyn Bridge in New York to test out the Cadillac’s new Driver Awareness and Driver Assist active safety technologies.  This includes the ability to maintain a safe following distance in stop-and-go traffic.  The Brooklyn Bridge’s metal design and traffic let GM engineers refine radar sensors so that they are able to tell the difference between stationary objects (guard rails and bridge structure) from actual vehicle traffic.

Metal structures often times confuse radars built into vehicles into thinking that an obstacle is approaching said GM’s lead engineer for Sensor Fusion Jim Nickolaou.  Nickolaou had previously worked on F14 and F15 fighter jets (check out our interview below).  Sensor Fusion is essential in the science of determining how data should be used when collected from cameras and sensors.  The technology is built into Cadillac’s all-new 2013 XTS luxury sedan and ATS compact sport sedan.

?The camera, sensors and radar technology act as the ?brain? behind all the safety features, feeding data 25 times per second into the car?s computer network,? said Nickolaou. ?We found that the best way to test the system?s accuracy was to gauge its performance in stressful driving conditions that could confuse it, like those conditions found on the Brooklyn Bridge.?

The Brooklyn Bridge is one of the largest suspension bridges in the world and it stretches across the East River, which separates Manhattan and Brooklyn.  The bridge has thousands of metal cables, barriers, and structures.  Since the bridge is regularly congested with New York City traffic, it helped GM really test out Sensor Fusion.

About 2,000 scenarios were identified to test sensors and radars including variables like traffic, weather, lighting, and radar reflections.  All of these variables could influence how radar systems and sensors work in the car.

?We have to try and anticipate every scenario, especially those that really challenge the technology,? added Nickolaou. ?We tune the systems to discern actual obstacles from other things in the vicinity that should not cause a warning or braking action.?

Driver Assist Package has other safety features like Automatic Collision Preparation, Front and Rear Automatic Braking, and Lane Departure Warning.

?We?re bringing the future forward for today?s drivers,? stated Don Butler, VP of Cadillac Marketing. ?The Driver Awareness and Assist technologies are tailored for intuitive use in the rigors of daily commuting.?

To gain a better understanding of the experiment, I interviewed Mr. Nickolaou.  Below is the Q&A:

1.a Coming up with the idea to test out Sensor Fusion on the Brooklyn Bridge is very creative.  How did the idea first originate?  

Cadillac, as part of the normal development of new technology, tries to expose the vehicles to what our customers would expect in every day driving. In testing of the new technology we have mapped out many different trips and development drives across the U.S. as well as some other global markets. They have totaled thousands of miles to date and we are increasing this sample size rapidly to ensure the technology and vehicles have properly been exposed to the rigors of every day driving.

Metal and radars are a natural challenge. In our testing plans, we wanted to have complicated routes and challenges the radars not only bridges, but in tunnels, windy roads, etc.  We used our experience and started methodically testing on the Brooklyn bridge and other bridges and tunnels across the United States.  To date, Brooklyn and Manhattan are still some of the most challenging major cities for the sensors due to the moving objects (vehicles, pedestrians, etc.) and shadow transitions in tunnels, overhead structures, buildings, etc.

Other cities we’ve tested are Chicago, San Francisco, Los Angeles, New Orleans, Boston, Atlanta, and Shanghai.

1.b. Was there any initial resistance from the New York City government for using the bridge on testing?

We made several phone calls and the NYC Precincts in Manhattan, Bridge Authority, and Port Authority.  All helped in making this testing go smoothly.  In fact, without their help we may have been pulled over and had our equipment confiscated.  NYC authorities alerted all their areas and our testing progressed smoothly.  A round of applause and a heartfelt thank you was sent afterwards to NY from our side as they (NYC) all seemed to work cohesively.

2. Can any of the Sensor Fusion technology be used to enhance GPS capabilities in cities that have high-rise buildings?

Sensor fusion is a generic term which basically means combining sensory data from a disparate set of sensors to produce resultant information better than an individual sensor.  In the case of GPS challenges in an urban canyons (i.e. tall buildings which shadow or cause multiple reflections of GPS signals) ? like NYC ? the location of your vehicle from vision and radars may help improve the understanding of the vehicle’s position. This technology is not yet available at the automotive consumer level. This level of technology is still being worked on in academia and automotive industries.

3.a. To my understanding, Sensor Fusion could often times “take action if needed” if the driver is in any danger such as accidentally merging into a lane without noticing a speeding car approaching from their blind spot.  What are some of the most challenging aspects in integrating this feature?

The feature you refer to where the vehicle ?takes action if needed? by accidently merging into another lane is called Lane Departure Warning.

We conducted significant research to learn human behaviors when under certain situations, like when you are backing up and an obstacle interferes unexpectedly. Gathering and analyzing this data so as to apply it to our customer’s benefit and safety is a long process, but well worth it, and necessary.

3.b. What are some of the telltale signals that Cadillacs have to warn people if there is a car in their blind side?  

Cadillac has several ways to alert drivers.  My favorite is the Safety Alert Seat, which vibrates the driver?s seat bottom if s/he begins to drift into another lane on a marked freeway or if there is an object approaching when the vehicle is in reverse.

Cadillac also has signals which flash in the outside mirrors indicating traffic in the blind spot (Lane Departure Warning).  Through CUE, the driver has the ability to set their warning preferences (Haptic or Haptic and Audio).

4. After collecting data from the Brooklyn Bridge experiment, what is the engineering process from getting to the point where you learn things from the experiment and then integrating it into a shipped feature built into the cars?  

Thankfully, Cadillac learned early in development opportunities to improve hardware and software. Working with sensor suppliers we were able to not only redo their hardware but work with them on improvements in software.  This, along with some unique patented algorithms in Cadillac’s processing modules, we were able to through simulation, testing, and continuous validation activities to integrate this into the production Cadillacs.

5. What are some of the key-takeaways from the Brooklyn Bridge experiment?

The biggest take away was the complexity of the sensing environment we needed to be immune to, not only from radar and vision but other sensors, as well as meeting the driver’s expectations in assisting when needed and on the day to day use of this vehicle.

6. Were there any major similarities between developing similar technologies on F14 and F15 fighter jets versus a Cadillac?

I think the complexity of air-to-air radar is very similar with today’s automotive technology.  The F14 and F15 fighter radars I worked many years ago tracked as many targets as automotive radars do today!  The complexity comes when you bring the radars closer to the ground and every crack in the road, debris, curbs, metal, etc., start reflecting energy from different paths and angles.

Just like a fighter jet (which also has a camera on it), Cadillac takes the advantages of the accuracy of the camera and attempts classification of vehicles versus other objects with the advantage of Doppler shifting in the radars.  Combining together (or fusing) we locate the moving and non-moving objects and try to recognize the driver’s situation to assist on the day-to-day drive by sending road data to the vehicle?s computer 25 times per second.

You may also like:

– How Much Is My Car Worth? Here Are Some Tools To Help.

– Best Electric Cars