Meta launches image generation model with coding, search capabilities

Meta Platforms Inc. today debuted an image generation model that can write code and search the web.

Muse Image is the second algorithm released to date by Meta Superintelligence Labs, the company’s artificial intelligence research group. The first is the Muse Spark large language model that made its debut in April. Both algorithms are available through the Meta AI chatbot.

Muse Image can generate images based on highly detailed, multi-sentence user prompts. It’s also capable of editing existing photos. Users can have the model remove elements such as fog, change the camera angle and perform other changes. The Meta AI chatbot provides the option to refine Muse Image’s output by uploading a sketch that explains what edits it should make.

Under the hood, the algorithm shares several similarities with reasoning-optimized LLMs. The perhaps most notable is that it features a tool use capability.

If a prompt doesn’t contain all the details that Muse Image requires to generate a file, it can use a search tool to retrieve the needed information from the web. Another built-in tool enables the model to generate code. For example, it could write a Python script to turn the contents of a spreadsheet into a graph. Scripts help increase the accuracy of complex visualizations.

Muse Image can loop in Muse Spark, Meta’s reasoning model, when its coding capabilities are insufficient to process a user request. The latter LLM is capable of turning images generated by Muse Image into websites and video games.

Many image generators refine their output using a method called best-of-N, or BoN. The technique consists of generating multiple media files and picking the one that best aligns with the user’s prompt. 

Muse Image takes a different approach. Meta says the model engages in “deliberate reasoning” before it starts generating an image. According to the company, that method enables the model to make better use of the underlying infrastructure than BoN. Muse Image supports a test-time compute feature, which means that increasing the amount of hardware at its disposal boosts output quality.

The model reviews the images that it generates before displaying them to users and makes tweaks when necessary. Meta says the model’s self-refining behavior, as it calls the feature, emerged on its own during the reinforcement learning phase of training. Reinforcement learning hones an AI model’s reasoning capabilities through trial and error.

On launch, Muse Image is available via the Meta AI chatbot in a limited number of markets. It also powers a set of new image effects in Instagram stories. Meta plans to bring the model to Facebook, Messenger and more parts of Instagram in the future.

Further down the line, the company will launch a clip generator called Muse Video. The current early iteration of the model is in third place on the popular Arena AI ranking of video generators.

Meta plans to make Muse Image available to advertisers through its Advantage+ suite of marketing tools in the coming weeks. The company will likely take the same approach with Muse Video. Given that Meta reportedly plans to launch an AI infrastructure service, it’s possible the two models will eventually become available to developers via an application programming interface. 

Image: Meta