https://typesense.org logo
Join Slack
Powered by
Hey guys, I'm having some issues trying to export ...
# community-help
g
Hey guys, I'm having some issues trying to export a collection + left join Let's say that I have a source_collection and a join_collection. 
join_collection
schema references 
source_collection
using the 
source_id
field My request is something like this in source_collection: filter_by: id:* || $join_collection(id:*) include_fields: $join_collection(*) So basically this request works fine for the multisearch endpoint, but the export just keeps running until it times out, and all the subsequent requests fails with 503 for some seconds. I suspect that this is breaking because not all documents in source_collection are referenced in join_collection. Multisearch handles this by doesn't showing the joined document, but the export just crashes.
j
Could you try with 
v28.0.rc36
? This sound similar to an issue we fixed recently
g
Didn't work, same problem 😕
j
Could you put together a set of minimal curl commands like this that replicates the issue and open a GitHub issue? We can then look into it
g
Sure
ok, can you try this? ### Run Typesense via Docker ######################################## export TYPESENSE_API_KEY=xyz mkdir "$(pwd)"/typesense-data docker run -p 8108:8108 \ -v"$(pwd)"/typesense-data:/data typesense/typesense:27.1 \ --data-dir /data \ --api-key=$TYPESENSE_API_KEY \ --enable-cors ### Reproduction Steps ############################################### export TYPESENSE_API_KEY=xyz curl "http://localhost:8108/debug" \ -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" curl "http://localhost:8108/collections" \ -X POST \ -H "Content-Type: application/json" \ -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \ -d '{ "name": "redemption_codes", "fields": [ {"name": "code", "type": "string" }, {"name": "created_at", "type": "int64" } ] }' curl "http://localhost:8108/collections" \ -X POST \ -H "Content-Type: application/json" \ -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \ -d '{ "name": "transactions", "fields": [ {"name": "redemption_code", "type": "string", "reference": "redemption_codes.code", "optional": true }, {"name": "amount", "type": "int64" }, {"name": "created_at", "type": "int64" } ] }' curl "http://localhost:8108/collections/redemption_codes/documents/import?action=create" \ -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \ -H "Content-Type: text/plain" \ -X POST \ -d '{"id": "124","code": "1234","created_at": 12345678 } {"id": "125","code": "5678","created_at": 12345678 }' curl "http://localhost:8108/collections/transactions/documents/import?action=create" \ -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \ -H "Content-Type: text/plain" \ -X POST \ -d '{"id": "124","redemption_code": "1234","amount": 100,"created_at": 12345678 } {"id": "125","amount": 200,"created_at": 12345678 }' # note that this transaction does not have a redemption code echo $"\n\nTrying to multi search\n" curl "http://localhost:8108/multi_search" \ -X POST \ -H "Content-Type: application/json" \ -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \ -d '{ "searches": [ { "q": "*", "collection": "redemption_codes", "filter_by": "id:* || $transactions(id:*)", "include_fields": "$transactions(*)" } ] }' echo $"\n\nTrying to export\n" curl "http://localhost:8108/collections/redemption_codes/documents/export?filter_by=id:*%20%7C%7C%20$transactions(id:*)&include_fields=$transactions(*)" \ -X GET \ -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
j
Could you put that in a text snippet in Slack or surround it by tripple quotes, otherwise Slack messes up the quotes and it's hard to execute it
g
Copy code
### Run Typesense via Docker ########################################
export TYPESENSE_API_KEY=xyz
    
mkdir "$(pwd)"/typesense-data

docker run -p 8108:8108 \
            -v"$(pwd)"/typesense-data:/data typesense/typesense:27.1 \
            --data-dir /data \
            --api-key=$TYPESENSE_API_KEY \
            --enable-cors

### Reproduction Steps ###############################################
export TYPESENSE_API_KEY=xyz

curl "<http://localhost:8108/debug>" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}"


curl "<http://localhost:8108/collections>" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
       -d '{
         "name": "redemption_codes",
         "fields": [
           {"name": "code", "type": "string" },
           {"name": "created_at", "type": "int64" }
         ]
       }'

curl "<http://localhost:8108/collections>" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
       -d '{
         "name": "transactions",
         "fields": [
           {"name": "redemption_code", "type": "string", "reference": "redemption_codes.code", "optional": true },
            {"name": "amount", "type": "int64" },
            {"name": "created_at", "type": "int64" }
         ]
       }'
       
curl "<http://localhost:8108/collections/redemption_codes/documents/import?action=create>" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -H "Content-Type: text/plain" \
        -X POST \
        -d '{"id": "124","code": "1234","created_at": 12345678 }
            {"id": "125","code": "5678","created_at": 12345678 }'

curl "<http://localhost:8108/collections/transactions/documents/import?action=create>" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -H "Content-Type: text/plain" \
        -X POST \
        -d '{"id": "124","redemption_code": "1234","amount": 100,"created_at": 12345678 }
            {"id": "125","amount": 200,"created_at": 12345678 }' # note that this transaction does not have a redemption code
            


echo $"\n\nTrying to multi search\n"
            
curl "<http://localhost:8108/multi_search>" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '{
          "searches": [
            {
            "q": "*",
          "collection": "redemption_codes",
          "filter_by": "id:* || $transactions(id:*)",
          "include_fields": "$transactions(*)"
            }
          ]
        }'


echo $"\n\nTrying to export\n"

curl "<http://localhost:8108/collections/redemption_codes/documents/export?filter_by=id:*%20%7C%7C%20$transactions(id:*)&include_fields=$transactions(*)>" \
        -X GET \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
j
CC: @Harpreet Sangar ^
h
@Gabriel Marinho The issue is with url encoding
Copy code
curl -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -X GET \
    "<http://localhost:8108/collections/redemption_codes/documents/export?filter_by=id%3A%2A%20%7C%7C%20%24transactions%28id%3A%2A%29&include_fields=%24transactions%28%2A%29>"
returns
Copy code
{"code":"1234","created_at":12345678,"id":"124","transactions":{"amount":100,"created_at":12345678,"id":"124","redemption_code":"1234"}}
{"code":"5678","created_at":12345678,"id":"125"}
So basically this request works fine for the multisearch endpoint, but the export just keeps running until it times out, and all the subsequent requests fails with 503 for some seconds.
Still, this shouldn't happen.
g
hey, thanks for the response yeah you're right. I changed that URL and it worked fine. Then I tried inverting the order of imports and it crashed the server with this error. This seems more likely to what is happening at our server. What's weird though is that the multisearch in that case isn't showing the joined field, while it shows it correctly on our server. Can you try this file?
typesense.sh
h
What typesense version are you running?
g
27.1
h
I was able to reproduce the issue. We'll include a fix for this scenario. You should mention 
async_reference: true
in your reference field definition if you can't control the order of the insertion.
g
Awesome, thank you. Please let me know when you open a GitHub issue for this, so I can be aware of any available fixes
h
Okay. The fix still won't help in your scenario though. It would only take care of the crash. You'll still receive no result when documents are indexed out of order.
g
Oh got it, so the real fix for me is the async_reference: true right?
✅ 1
h
Also, to add the 
async_reference: true
property, you'll have to reindex your collection since we don't yet support altering the 
reference
field.
g
Great, I'll take a look. Thank you
🙌🏼 1
Hey @Harpreet Sangar, can I run this typesense.sh in rc36? I already switched my prod typesense to rc36, added async_reference and reindexed my collections, but still the export isn't working :/
h
Yes it should work even with v27.1
redemption_code
should look like this
Copy code
{
  "name": "redemption_code",
  "type": "string",
  "reference": "redemption_codes.code",
  "optional": true,
  "async_reference": true
}
g
Oh ok. Yeah the curl commands are working with async_reference. I'll check if I'm reindexing it correctly
✅ 1
Hey @Harpreet Sangar, we were chatting with Jason via email about this, he asked to continue the conversation here again. So basically the export with left join still isn't working yet even with "async_reference" in our schemas. We tried the new rc37, which at least stopped the cluster to crash when we do this request, but still we can't get the joined fields in response. When we try the multisearch with the same request, it works perfectly. I know that I could make it work with the curl commands, but in the cluster we can't get it to work.
h
Can you share the multisearch request?
g
Copy code
- q:*
- filter_by: id:* || rockitcoin_transactions_kiosk(id:*)
- include_fields: $rockitcoin_transactions_kiosk(id)
export is the same without the "q" field
h
Hi @Gabriel Marinho Got pulled into other tasks sorry couldn't reply any sooner. I'll try to reproduce it on Typesense Cloud.
> but in the cluster we can't get it to work. Are you using Typesense Cloud or self hosted?
g
Typesense Cloud + the javascript client
h
Okay
Can you share the javascript client code you're using for export?
g
Copy code
const res = await typesenseClient.collections("rockitcoin_transactions_kiosk").documents().export({ filter_by: "id:* || $rockitcoin_transactions_kiosk(id:*)", include_fields: "$rockitcoin_transactions_kiosk(id)  });
If you don't mind, I sent a video to your DM showing the requests in my browser's network tab
✅ 1
h
I was finally able to find the bug. https://github.com/typesense/typesense/pull/2217 fixes it.
Copy code
const res = await typesenseClient.collections("rockitcoin_transactions_kiosk").documents().export({ filter_by: "id:* || $rockitcoin_transactions_kiosk(id:*)", include_fields: "$rockitcoin_transactions_kiosk(id) as transaction });
will work as expected now.
g
Thank you, this is awesome!
So this will be available only in v29?
h
I'm not sure yet.
@Kishore Nallan can tell you about the release.
k
Will share an updated RC build later today.
g
Nice, thanks guys
k
We've published 
29.0.rc2
g
Just confirmed that it did work for us!
🙌 1
🙌🏼 1
2 Views
Open in Slack
PreviousNext
Powered by