Bay Area engineers work to cram AI into smart hardware gadgets

At a Fort Mason workshop with sweeping views of the Golden Gate Bridge, most of the tinkerers inside rarely look up to take in the stunning setting.

Instead, on the second floor of tech incubator Founders Inc., more than a dozen young inventors sit hunched over lines of code or prodding wires into circuit boards hoping to build the next great electronic gadget powered by artificial intelligence.

Scattered around a lab strewn with wires, industrial drills and battery components are devices in various stages of creation or repair, ranging from smart goggles that can interpret the world for blind wearers to a handheld device that would help humans navigate doomsday scenarios.

Worlds apart from software chatbots that converse like humans, answer queries or create images out of thin air, these experiments are an effort to cram AI into hardware and create smart gizmos that interact with the surrounding world in ways never before possible.

Founders Inc., which spots and funds promising innovations and startups, invited more than a dozen tinkerers from across the world to spend six weeks at its airy Fort Mason hardware lab starting in mid-April. The best ideas stand to receive six-figure seed capital investments.

It’s no wonder entrepreneurs and financiers are chasing the opportunity. Market analyst Precedence Research estimates that the market for electronics containing AI has already hit more than $50 billion this year — including uses such as computer chips, storage and networking, across all end markets — and could reach nearly $500 billion in a decade.

Some of the early attempts at AI-powered consumer devices, such as the Ai Pin from San Francisco-based Humane Inc., haven’t taken off — likely because they were delivered to market prematurely. Founders’ Head of Capital, Safwaan Khan, is looking for the innovations that stick.

“It’s not just about building the hardware,” he said. The residency is pushing builders to learn from the smart people around them to speed up problem-solving.

One of those inventors still getting the kinks out of his machine was Shubh Mittal, easy to pick out in chunky black glasses with a built-in camera perched on the bridge of his nose above a wide grin.

Paired with AI programs such as OpenAI’s ChatGPT and Anthropic’s Claude, Mittal’s camera can scan images and text and describe them in a computerized voice in about three seconds. “The idea is to help blind people be more independent,” by using the glasses to read menus and describe their surroundings, Mittal said.

But getting complex AI software to play nice with evolving hardware can be extremely tricky.

Case in point: When Mittal recently demonstrated his latest software update in front of the assembled cohort — a Friday ritual to highlight the week’s progress — the results were mixed.

Asking the glasses to identify a pink stuffed elephant in front of him, at first Mittal got no response. Finally the program denied the existence of the elephant, describing instead a forest scene entirely unrelated to the hardware lab.

A slightly bashful Mittal said moving his head had created a blurred image that confused the program. Later, the glasses described the lab with uncanny accuracy, identifying it in detail as a research or industrial area.

The Founders lab and Mittal aren’t the only people working on smart glasses. Engineers at the SRI International research center in Palo Alto have sunk millions in government grants into head-mounted AI technology that could one day help untrained soldiers dress a battlefield wound or fix a knocked-out engine. The goal is to create a headset that uses AI to talk users through steps using a visual display and spoken instructions.

Those plans are a ways off. For now it can help you make lunch.

Wearing a Microsoft HoloLens headset tethered to a bank of computers, SRI researcher Bob Price recently demonstrated how the device can teach someone to make a tortilla wrap.

The headset requires a user to push virtual buttons in the air that are projected onto the heads-up display. As the robotic voice walks a user through the steps, built-in cameras can reliably detect when the tortilla has been smeared with peanut butter and jelly, rolled and sliced.

Project supervisor Charles Ortiz said the hardest part is training the system to recognize mistakes and respond appropriately. Even a user’s hands getting in the way can confuse the machine.

Another project is happening a few miles away at a Stanford University lab. A team led by Gordon Wetzstein, an associate professor of electrical engineering who runs the school’s Computational Imaging Lab, is trying to perfect glasses that display 3D imagery on spectacles lighter and sleeker than the clunkier HoloLens.

Wetzstein used algorithms and laser etching to model how light projected from a side-mounted source would bounce around the lenses and into a user’s eye, creating a 3D hologram effect.

Instead of headsets that project flat images, Wetzstein hopes the AI goggles of the future will look more like his lighter device, which uses tricks of light to create augmented reality.

“The convergence of AI and hardware really enables new things,” he said.

While glasses and headsets are an obvious early application for AI hardware, the tinkerers back at the Founders lab are chasing even more esoteric opportunities.

Engineer Adam Cohen Hillel is fine tuning a product he calls Ark, a handheld device meant to help users survive Armageddon. 

On the assumption that networks and communication would be destroyed, Ark is preloaded with maps and uses Meta’s Llama 3 AI engine to tell users where to find water or shelter, or how to administer first aid or start a fire.

Right now, it’s a screen on a circuit board fitted with a stylus that Cohen Hillel jokes kind of resembles a makeshift bomb. 

During his demonstration in front of the group, the map moved smoothly, but the voice chat function went down. Cohen Hillel hopes to lose the stylus in the future to run the device solely on voice commands and add a solar panel to power the Ark when the grid is gone. He’s also planning to add a special case so it can withstand an electromagnetic pulse from a nuclear bomb.

“Cramming that much data on a small board is the hardest part,” he said. So is getting it to work fast enough.

But when they do work, devices like Mittal’s and Cohen Hillel’s offer a glimpse of the future. 

Getting to a “magic moment” where something works is the easy part, said Hubert Thieblot, general partner at Founders Inc. Getting something to work every time is much harder.

Reach Chase DiFeliciantonio: chase.difeliciantonio@sfchronicle.com; Twitter: @ChaseDiFelice