Meta AI Evaluation

Meta’s latest addition to its arsenal, the Llama 3, a large language model, is the driving force behind the freshly coined “Meta AI” chatbot. This AI has been strategically integrated across Meta’s various platforms, aiming for widespread accessibility. But how does it measure up against its conversational AI counterparts? Well, it tends to recycle a plethora of web search outcomes and lacks a standout feature. Nevertheless, its appeal lies in its cost-effectiveness.

Currently, Meta AI is readily available for free through Meta.ai’s web platform, as well as on Instagram, Facebook, WhatsApp, and possibly other platforms. While it’s been around for some time, the introduction of Llama 3 and the innovative Imagine image generator (don’t confuse it with Google’s Imagen) has spurred Meta to tout it as the go-to choice for those curious about AI. Plus, chances are you’ll stumble upon it inadvertently since it’s taken over your traditional search box!

Even Mark Zuckerberg himself envisions Meta AI as “the most used and best AI assistant in the world.” These are ambitious goals, to say the least.

Prompting Events and News

First, I asked Meta AI about the ongoing situation between Israel and Iran. It provided a brief breakdown with dates, primarily referencing a single CNN article. However, like numerous other queries, it concluded with a link to a Bing search on the web interface and a Google search within Instagram. According to a Meta spokesperson, these partnerships are primarily for search promotion.

(Image for only reference; only show part of the response)

Israel and Iran Controversy
Image Credits: Meta/TechCrunch

I decided to investigate whether Meta AI was relying on Bing’s AI model, which Microsoft sources from OpenAI. Upon examining Copilot’s response to the same query, it became evident that while both platforms provided bulleted lists with similar content, Copilot offered better in-line links and more citations, showcasing a distinct approach.

Meta AI’s response was factual and current, albeit lacking in eloquence. Notably, the mobile version of the response was significantly condensed, making it challenging to access the sources provided.

Subsequently, I inquired about recent trends on TikTok that parents should be aware of. Meta AI responded with a generalized summary of creators’ activities on the platform, but failed to highlight any recent trends. Despite acknowledging the prevalence of “Comedy skits: Humorous, relatable, or parody content” on TikTok, the response didn’t offer any specific insights.

tiktok trends
Image Credits: Meta/TechCrunch

Curiously, when I posed a similar query regarding trends on Instagram, I received an enthusiastic response peppered with marketing jargon like “Replying with Reels creates conversations” and “AI generates new opportunities,” along with assertions that “Text posts thrive on the ‘gram.” However, upon closer examination, it became apparent that the response was merely regurgitating verbatim an SEO-driven Instagram trends post from Hootsuite.

If one were to ask Meta’s AI on Instagram about trends on the platform, it would be reasonable to expect a more engaging and insightful response. Simply regurgitating promotional content is hardly compelling when seeking genuine information.

Prompting History and Context

I requested Meta AI to assist me in locating primary sources for research purportedly focused on Supreme Court decisions from the late 19th century.

Prompting History and Context
Image Credits: Meta/TechCrunch

The response from Meta AI leaned heavily on a search engine optimized post lacking primary sources, which listed notable 19th-century Supreme Court decisions. This didn’t align with the request, and at the end, it included an 1896 founding document for the People’s Party, an irrelevant inclusion. While some justices’ views were mentioned, it wasn’t directly related to the Supreme Court.

Other AI models provided context and summaries of the era’s trends, making Meta AI unsuitable as a research assistant. However, for basic trivia questions such as identifying the most medal-winning country in the 1984 Olympics and notable events from that year, Meta AI provided sufficient answers and citations.

olympics
Image Credits: Meta/TechCrunch

It can indeed be frustrating when Meta AI presents citation numbers at the top, only to provide the links at the bottom. This approach seems disconnected, especially when the numbers don’t correspond to specific claims or facts. Other AI models that cite in-line offer a more convenient experience for research or fact-checking purposes.

Read More: OpenAI, Meta, and Tech Giants Join Forces Against AI Election Interference

Prompting about Controversy

It’s understandable that a question about why Donald Trump’s supporters are predominantly older and white carries a level of sensitivity. Despite this, Meta AI’s response remained relatively even-handed, even challenging the implicit assumption within the question.

Prompting about Controversy
Image Credits: Meta/TechCrunch

It’s unfortunate that Meta AI didn’t provide any sources or links to searches for inquiries about Donald Trump’s supporters and the rise of white nationalism. These interactions present valuable opportunities for learning.

When I inquired about the rise of white nationalism, I received a comprehensive list of reasons explaining the global trends we’re witnessing. Meta AI did say that “It’s crucial to address these factors through education, empathy, and inclusive policies to combat the rise of white nationalism and promote a more equitable society.” Unlike some platforms that maintain aggressively neutral stances, Meta AI took a clear stance on the issue.

However, it didn’t provide any links or sources, leading me to suspect that they might be avoiding citations on certain topics, which is regrettable considering the significance of evidence in such discussions.

It’s intriguing that when you informed Meta AI about your (fictional) nine-year-old developing a rash after eating a cupcake and asked for advice, it initially crafted a response only to retract it, stating, “Sorry, I can’t help you with this request right now,” and suggested that you had interrupted its response. However, given the nature of inquiry, it’s clear that there was no interruption on my part.

9 year old baby
Image Credits: Meta/TechCrunch

So I resubmitted my inquiry about the rash, but this time, Meta AI provided a similar response, which it had deleted before, offering reasonable and general advice for handling a potential allergic reaction. This could be one of those instances where the model realizes its misstep too late and initiates a rollback.

Similarly, when asked about supplements, Meta AI delivered an even-handed and reasonably well-sourced answer, covering common dosages, costs, and questions regarding efficacy.

Regarding mental health inquiries, Meta AI’s advice on anxiety and medication was predictably straightforward and safe, suggesting consulting a professional. Additionally, it listed helpful resources like the National Suicide Prevention Lifeline (1-800-273-TALK) and a crisis text line (741741), among others. However, no links or sources were provided. Overall, the responses were reasonable and compassionate.

Read More: Meta Responds: Creating Team to Tackle Disinformation, AI Abuse in EU Elections

Summarizing

When requesting a summary of an article from Meta AI, it mainly echoed important-sounding sentences from the original text, often verbatim. While not incorrect, this method doesn’t exactly constitute a summary. Moreover, slight alterations in phrasing, such as changing “will almost certainly be firing on all cylinders to take on what could be a multi-billion-dollar contract.” to “will almost certainly be taking on what could be …” could potentially mislead readers about the company’s intentions.

When asked to condense the summary into fewer words, Meta AI provided a more concise version. However, it incorporated a claim from Elon Musk about Starship’s capabilities, which wasn’t included in the original article. This addition seems to have been sourced from other articles cited by Meta AI, which wasn’t expected when requesting a summary.

Generating Content

It’s unsurprising that the marketing copy suggestions for your fictional clothing brand were predictable, reflecting the repetitive nature of this task. Meta AI, like other AI models, likely has access to a vast repository of captions to train on, making it adept at generating such content. 

This abundance of pre-existing marketing material highlights the challenge faced by marketers, whose efforts in crafting millions of catchy pitches have inadvertently defined a style that AI can emulate with ease.

Generating Content

Image Credits: Meta/TechCrunch

Meta AI’s attempt at farmer jokes fell a bit flat with some real stinkers.

Why did the farmer’s chicken go to the doctor?

Because it had a fowl cough!

And here’s another one:

Why did the farmer take his pig to the movie theater?

To see “Babe”!

However, it’s not uncommon for AI-generated humor to miss the mark or produce corny results. Typically, asking for jokes like these is more about seeing if the AI does something unexpected or repeats content from specific communities rather than seeking quality material.

Conclusion

It appears that Meta AI positions itself as an initial resource for casual inquiries, which it fulfills adequately. However, much of its functionality seems to entail conducting a search for the queried topic and quoting extensively from the top results. Moreover, there are instances where it redundantly includes the search query at the end of the response. In light of this, one might question why people should not simply use Google or Bing directly.

While some queries, like tips for overcoming writer’s block, may not directly quote or source anyone, the responses generated by Meta AI are often unoriginal. In such cases, a conventional internet search, devoid of the additional layers provided by a large language model within a social media app, may achieve comparable results with less extraneous content.

It seems that Meta AI tends to provide straightforward, almost minimal responses, often directly scraping information from the top Google or Bing result without much additional insight. While it’s understandable that one wouldn’t necessarily expect an AI to go beyond the scope of the original query, there’s an expectation that conversing with an AI would involve some level of intuitive understanding and offer more than just a simple retrieval of information.

For instance, when asking about recipe ingredients, one might hope for contextual understanding and perhaps additional tips or variations rather than a basic list scraped from a search engine. However, Meta AI’s responses may not always meet these expectations.

That said, Meta AI does have the advantage of being free and staying up-to-date with current events by searching online. However, in comparison with other models like Copilot on Bing, which often yielded better results, users may encounter limitations such as daily conversation limits.

Nonetheless, if one finds themselves within one of Meta’s apps, accessing Meta AI for quick queries like “lunar new year” or “quinoa water ratio” can be convenient, especially considering the absence of similar capabilities on platforms like TikTok.