Errors with Archie

Hello ​@Reckless_Kelly 

I hope you used the feedback button to let us know there were issues.  I am also going to turn this into a support ticket for our team to look into.

Can you tell us more about how the feedback feature works? 

Hello Alan,

I am checking in with one of our Development team ​@stevesaul regarding how the feedback is used.

Thank you. I use Archie daily and find it a tremendous timesaver. But I don’t know its limitations. Archie can prepare a list of files (if I select them), but what’s the limit on the number of files? Archie can inspect a document and tell me about what’s in the document - but for a large PDF, it doesn’t review all of the file? What’s the size limit? Archie can review all of the files in the matter looking for information (for example, “Find all documents that reference taxes), but it can miss a lot of information. What are the limitations on these types of searches? What are the character limits on what I can paste into the archie question? For example, if I want to provide archie with a list of a few rows from a spreadsheet and sort them by date or amount - there’s a limit to what I can input in the dialogue box - what is the limit?

​@Funkytown ​@Reckless_Kelly 

Can you tell us more about how the feedback feature works? 

All feedback whether thumbs “Up” or thumbs “Down” is reviewed on a daily basis by the Archie Product team. This team can look at the Question that was asked in the prompt and the comments you, the user, provides in the feedback field. From this we try to understand the problem/challenge without being able to see what the matter data/document contained or the response was.  

From this we work with the developers on tweaks to how data is extracted from documents, how we search for relevant information on the matter relating to the question, and how we interact with the LLM (AI large language model).  

We release new improvements on a monthly basis and the feedback has been very beneficial in understanding how people use Archie and where and why it falls short (The more information in the comments on thumbs down the better!). 

​@Funkytown 

Archie can prepare a list of files (if I select them), but what’s the limit on the number of files? 

Today, this is not a great use case for Archie as it is unlikely that you will receive an accurate answer and it is limited to as few as 4 files.  

We are actively researching in this area to extend Archie with some specific functions like you mentioned above, which looks to be a type of “Discovery” implementation of AI. This will be looked at in the first few months of 2025 and will allow Archie to do a better job of this use case for you.

​@Funkytown 

Archie can inspect a document and tell me about what’s in the document - but for a large PDF, it doesn’t review all of the file? What’s the size limit?  

We do not have a hard cutoff in terms of “Pages” as it is a text (characters) limit. This limit we estimate to be around 180 full pages of full text, which is quite a lot.  

​@Funkytown 

Archie can review all of the files in the matter looking for information (for example, “Find all documents that reference taxes”), but it can miss a lot of information. What are the limitations on these types of searches?  

When Archie is asked to look for an answer across all documents on a matter, we utilize a technology called Semantic search using AI.  

Semantic search is a more advanced way of finding information that focuses on understanding the meaning or intent behind a search query rather than just matching exact words or phrases. Instead of relying solely on keyword matching, it uses AI and natural language processing to interpret the context, relationships, and concepts in both the query and the data being searched. 

This works very well in a lot of cases, but there can be limitations based on a number of factors. This includes  

• The prompt/question itself (how specific it is – the more specific the better) 

• The number of documents on the matter (the more documents there are, that may contain relevant information, the harder it is to get all of that information and not miss any) 

• The quality of the documents (ie how we can extract the text, Word is easy but PDF can be tricky in some cases using OCR). 

We continue to refine Archie to do a better job in this area, as well as teach users how to use it in the best possible way. We believe Archie will become better in this area over the next 3-6 months and the more feedback if it is not accurate, the more it helps us refine it.

​@Funkytown 

What are the character limits on what I can paste into the archie question? For example, if I want to provide archie with a list of a few rows from a spreadsheet and sort them by date or amount - there’s a limit to what I can input in the dialogue box - what is the limit? 

The prompt box has a limit of around 90k words.  

In this specific use case, Archie does not do too well with bulk spreadsheet data. A few rows should be fine, but if it has hundreds of thousands of rows this is not a good use of Archie at this time.   

Add in the data to a Microsoft word document on the matter might actually produce a better result – you could give that a try! 

​@adeleh - thank you, these are super helpful responses. 

Question: Is there a way for Archie to notify the user when the response is inferred through the use of metadata without having to ask specifically?  See the following example:

Today- I learned to be aware of Acrhie using metadata in its responses: 

Where did you get information that Kara made an email complaint on December 6, 2024?

RESPONSE:

The information regarding the email complaint from Kara Stimpson comes from the document titled "Email Complaint" (1). This document outlines her formal complaint regarding discriminatory termination, harassment, and retaliation that she witnessed and experienced.

The specific date of December 6, 2024, is inferred from the metadata of the file, which indicates the date it was last modified. The content within this document and corresponding metadata support the timing and nature of her complaint.

​@ELS410eg I’m sure you have it covered but out of an abundance of caution, does that name need to be anonymized?