Hey all, I have a question about reducing RAM in my cluster. Basically, I have this column with a massive text field in it. I have been running a script to paginate through the collection and drop the field on every document. I thought with the size of this field, we’d see ram reduce pretty significantly. However, I have not seen it decrease at all. Is there anything I should trigger to return that ram? The plan was to basically drop this giant field, see what we’re saving, and then use the savings to fill out a new slimmed down collection, but without seeing a decrease in ram, I’m a bit concerned. Thank you as always for the amazing help!

Could you try updating the collection schema to drop the field entirely? Other than that, you could create a new collection without that field and see how much RAM that takes up as well

@Fanis Tharropoulos - I triggered that last night around midnight CST. It went smoothly. There are still some 2 or so million documents I need to paginate through and drop.

How are you dropping the field from the documents?

Pretty basically:

while (true) {
    const searchResults = await client
      .collections<{id: number | string}>(COLLECTION_NAME)
      .documents()
      .search({
        q: '*',
        query_by: 'id', // or another indexed field
        page,
        per_page: perPage,
      });
    if (!searchResults || !searchResults.hits) {
      console.error('Failed to retrieve search results:', searchResults);
      break;
    }
    console.log(`Processing page ${page}...`);

    if (searchResults.hits.length === 0) {
      break;
    }
    await client
      .collections(COLLECTION_NAME)
      .documents()
      .import(
        searchResults.hits.map(h => ({
          id: h.document.id,
          [FIELD_TO_DROP]: null,
        })),
        {
          action: 'update',
        }
      );

    if (searchResults.hits.length < perPage) {
      break;
    }

    page++;
  }

You should look into this: https://typesense.org/docs/28.0/api/collections.html#update-or-alter-a-collection

I did that via the cloud UX last night at around midnight. However, I saw this tip:

TIP

Dropping a field affects only the schema and the in-memory index; it does not remove the data associated with that field from the existing documents stored on disk. In Typesense, documents can contain additional fields not explicitly defined in the schema.

To permanently remove a field and its data from a document, you must update the document directly, by passing null to the field you want to remove.

Maybe there’s something I’m misunderstanding about how ram is consumed. These fields aren’t faceted. Do they perhaps only consume disk space and not ram?

The second part of the tip is about disk usage, not RAM. Just dropping the field from the collection schema should free up RAM associated with that field

Very interesting! Thank you for taking your time to explain so much to me! I will take this back to my team and we will discuss next steps at standup. Again, thank you very much!!

Feel free to reply to this thread if anything else arises!