#community-help

Adding New Fields to Items in Typesense Without Wiping Data

TLDR Alex wanted to add more fields to their Typesense items without wiping data. Kishore Nallan explained they had to create a new collection instead, and suggested using automatic schema detection for future changes. They also discussed potential downsides of indexing every field.

27mo

Join the chat

Jun 02, 2021 (27 months ago)

Photo of md5-faf0fdba0b6739a6706f05c15b6738c6

Alex

08:50 AM

Hey. We would like to add some more fields to our items in typesense. We also want these fields indexed. Our typesense server is running in production, so we don’t wanna wipe all data to make this change. Is it possible to add new fields (indexed ) to the schema and our items on the fly?

08:56

Alex

08:56 AM

From what i can read from similar asked question, we have to make a new collection, is that the only way? This means, we must upgrade our Typesense Cloud memory to do this. We would like to have no downtime for this change.

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

08:56 AM

👋 Alex We don't have a way to modify the schema at the moment but that's on our backlog. You can just create a new collection and swap to it. The Collection Alias features makes this really easy to do without updating your app configuration: https://typesense.org/docs/0.20.0/api/collection-alias.html

08:59

Kishore Nallan

08:59 AM

For now, if your cluster is also not having enough memory, then you have to index on another cluster and swap over.

For future use: you can use automatic schema detection to help with schema changes. For e.g you can have a collection with a field defined as:


```{"name": ".*_int32", "type": "int32"}```
This way, you can easily add a new field called

age_int32` and this will be automatically be inferred and indexed by Typesense.

09:00

Kishore Nallan

09:00 AM

Automatic schema detection is available on v0.20.0

Alex

09:08 AM

Thanks Kishore!
Do you see any downsides to adding:

{"name": ".*", "type": "auto" },
{"name": ".*_facet", "type": "auto", "facet": true }

to our schema? To make sure we never have to swap clusters again.

Kishore Nallan

09:12 AM

The only downside is that every field present in the document will be indexed. Using something like .*_int32 gives you flexibility. If you have two integer fields only one of which must be indexed, then suffix the indexable field name with _int32 but for the other field name don't do that, so it is only stored and not indexed.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Discussions on Typesense, Collections, and Dynamic Fields

Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.

31mo

Updating Collection Schema and Querying Unspecified Fields in Typesense

Stefan asked about updating a collection schema. Kishore Nallan stated it's not currently possible but suggested automatic schema detection in version 0.20. Rishabh sought a workaround and Jason suggested creating a new collection with a new schema and re-indexing the data. Rishabh also asked about defining custom field IDs which Jason confirmed is not possible with Typesense.

29mo

Updating Collections Strategy and Faceting New Field

Nithin asked about strategies for updating collections and faceting new fields. Kishore Nallan suggested creating another collection, indexing in the background and using aliases to switch live traffic over, and shared details about the upcoming release.

29mo