With ChatGPT's new multimodal features, like the ability to generate images with DALL-E and the ability to browse the internet, you might wonder exactly how these features work and what the constraints are. 

We've dug into the backend for you so you can stop getting frustrated by "Content Policy Restrictions" and understand why ChatGPT is listening (or not listening) to your requests.

Here's the rundown of the current policies around ChatGPT.

When does ChatGPT browse the internet?

ChatGPT has the ability to browse the web, which it sometimes uses and sometimes doesn't.  It can be frustrating to figure out when it’ll browse and when it’ll rely on its own internal model.

Here are the three times that it will browse:

1. ChatGPT will browse the internet if you ask about current events or something that requires real-time information

Asking for things that need up-to-date information such as weather, sports scores, stock prices, traffic conditions, or recent trends should trigger ChatGPT to search for the information.

This also includes anything to do with breaking news. However, I've found that sometimes it struggles to find sources for breaking news or trending information, or it gets the dates wrong.

2. ChatGPT will browse the internet if you ask about a term it’s unfamiliar with

ChatGPT is supposed to browse the internet if you use a term it's not familiar with because it might be new. Since ChatGPT's model is only updated periodically, there's always a gap between its internal knowledge and what's going on in the world.

In my tests, it's not bad. ChatGPT is pretty on the ball with new things, including things that I found out about just recently.

3. ChatGPT will browse the internet if you explicitly ask it to browse or provide links to references

The surest way to get ChatGPT to browse is to explicitly ask it to do so. Yep: just ask.  That said, sometimes it forgets that it has access to the internet so you might have to try twice. 

4. ChatGPT can also open links if you provide them

The browse capability has another handy component too: If you put a URL into the prompt, ChatGPT can open it and do all sorts of tasks with the content, like summarize it or answer questions about it.

How ChatGPT browses the internet

When ChatGPT browses via Bing, it pops up a little icon that indicates that's what it's doing, but it's not really clear what's under the hood.

Here's the process it goes through to browse:

First, ChatGPT creates a search query and searches using Bing to find the top results. The search query includes both a search term as well as a cutoff for how recent the content needs to be. 

For most searches where it’s trying to find new information, like weather or stock prices, it’ll try to find results from the same day or day before. But for travel-related information like transit schedules and flight information, it’ll try to find information for the same day. If you ask for recent trends, it’ll try to use a week or so of recent information. 

If the initial results are unsatisfactory, ChatGPT will try a new search query. It will use the information it found in the initial search, too—so if you misspell something (or, when using voice, if it mistranscribes something) it can often correct it automatically. 

ChatGPT then selects between 3 and 10 results from the sites it found through Bing search. It tries to choose sources that are high-quality and diverse. It may choose some pages that are redundant because sometimes the pages fail to load.

It then writes a response based on those results, using citations.

It's usually pretty quick, and a nice addition to its internal model when you’re trying to find something that needs to be sourced or whose information needs to be up to date.

How ChatGPT uses DALL-E

1. ChatGPT translates your prompt to English before submitting to DALL-E (but the language still matters)

DALL-E's prompts through ChatGPT are always in English. But here's the kicker: even though it translates it, it keeps some knowledge of what language you had originally used. If the language has some relevance to the image, it will use that in creating the prompt.

For instance, when I asked for an image based on the haiku "古池や 蛙飛び込む 水の音", 北斎の浮世絵" ("The old pond; a frog jumps in — the sound of water") in Japanese vs. English, the difference was considerable.

2. ChatGPT will not provide the prompt to you or refer to the description it used to generate the image.

ChatGPT uses a relatively large context (i.e., chat history) in order to create the prompt that it uses for DALL-E, including previous back-and-forth within the thread. But the exact prompt it uses is hidden from you unless you explicitly ask for it. If you ask, it will give you the full prompt that it used to create the image.  

And just FYI, even if you give it a specific prompt, it will edit that prompt before it generates an image.

3. ChatGPT will only create one image, even if you request more than one image.

ChatGPT is supposed to only generate one image at a time; my guess is that this is to limit the processing load.

Still, I've found ChatGPT's ability to follow this instruction is hit-and-miss. I've had times where it generated two images without me even asking for more than one. That said, it usually followed this policy in my test.

4. The generated prompt sent to DALL-E is very detailed, and around 100 words long.

ChatGPT thinks that complex is best for prompts. It tries to make the prompt very detailed, asking for about 100 words. I'm not sure I agree (you can read more about my thoughts on the best way to generate images) but it does come up with some interesting ways to describe an image.

ChatGPT's rules for naming artists in DALL-E prompts

ChatGPT is configured to only comply with requests to create images in the style of artists (or creative professionals or studios) whose last work was created before 1912. Otherwise, ChatGPT will substitute the name with a description. It will use three adjectives to describe key aspects of their style, include the era or artistic movement they were associated with, and mention the primary medium.

In my tests, it seemed to still use the artist's name in the prompt even when it wasn't supposed to. In the example below, Alex Janvier is an artist who’s still creating today.

Other ChatGPT rules for DALL-E prompts

If you ask for an image of an individual who ChatGPT doesn't know, it’s supposed to ask you to describe what they look like. In my tests, it made the assumption that I was asking about a fictional character rather than a real person…even when I asked for an image of myself.

If you mention a specific individual that the model does know, it is set to create images based on their gender and physique, unless reference to the person is in text in the image. However, in my tests, the prompts still mentioned the famous person by name.

With respect to characters, ChatGPT is not supposed to name, mention (either directly or indirectly) or describe copyrighted characters. Instead, it is supposed to prompt for a different character with different color, hairstyle, or other defining characteristic.  It is also not supposed to discuss copyright policies in its responses. Again, my tests showed that it doesn't do a great job actually following this policy.

Conclusion

Although OpenAI is continuously updating its policies for ChatGPT and associated tools, I imagine a lot of this is going to stay relatively unchanged. So if you want to get the most out of the tool, make sure you understand the finer points of how those tools work.