Building AI That Knows What to Remember and What to Forget

Most conversations about artificial intelligence (AI) memory focus on how to store more data, such as longer context windows, larger vector stores, and retaining more history. However, when it comes to a personal AI assistant, the more challenging and significant question is determining what should not be kept and what should fade over time. This article presents a design note exploring the trade-offs involved in managing personal memory.

The idea of retaining every message permanently may seem safe at first glance. Yet, in reality, it leads to several issues. Noise accumulates due to the volume of stored information, making it harder to differentiate between relevant and irrelevant data. As more data is stored at equal weight, it competes with important information during retrieval.

Stale facts persist, with old addresses, cancelled plans, and outdated preferences still matching queries. Furthermore, sensitive data accumulates, requiring constant protection, exportability, and deletability.

A straightforward way to approach what should be kept is by sorting incoming information based on its future use:

1. Type: Commitments

Example: "Remind me to send the invoice on Friday"

Handling: Keep until the task is completed, then archive the reminder.

2. Type: Durable facts

Example: Someone's birthday, personal preferences

Handling: Keep these facts and allow updates to supersede older information.

3. Type: Reference material

Example: Saved articles, receipts

Handling: Keep these items retrievable but give them low priority in recall.

4. Type: Ephemeral chatter

Example: "Ok, thanks," small talk

Handling: Do not store this information as part of the memory.

The distinction between short-term and long-term memory is crucial. The short-term memory covers immediate context, which should expire quickly. Long-term memory retains commitments, durable facts, and reference material, which can remain in the system for extended periods.

When facts change, a memory system should be capable of replacing outdated information. For instance, if the user updates their dentist appointment, the old time should not coexist with the new one. Instead, the old record should be marked as superseded, enabling it to stop competing in retrieval while still being auditable.

Rather than assigning importance during the write time, signals can accumulate over time:

1. Was the information mentioned again?

2. Was it retrieved and utilized?

3. Did the user provide corrections or confirmations?

4. Is there a recurring date or person associated with the information?

As a result, less-used information can decay in priority, while frequently accessed data can be promoted.

User control over forgetting is essential for personal data. The system alone should not decide to erase sensitive information. Users should be able to view stored data, make edits, and permanently delete information. Two practical guides provide insights on deleting data from AI services and an AI data portability checklist.

Another useful approach for filtering information is entity extraction, which identifies people, places, and dates. Structured extraction can act as a relevance filter, as messages containing dates and people are likely to be worth keeping, while simple greetings are not. This technique was discussed in the context of AI extracting people, places, and dates from notes.

In conclusion, building good personal memory requires careful curation rather than hoarding data. Keeping commitments and durable facts, expiring context quickly, allowing updates to supersede old values, and providing users with control over their data are key principles in this design. The article highlights these principles as part of building a personal AI for reminders, lists, and memory across various communication platforms, such as WhatsApp, email, web, and desktop.