How to apply this to JSON

[LiveNathan]

spring-ai-examples/vector-store-loader/src/main/java/com/example/vectorstoreloader/VectorStoreLoaderApplication.java

Line 58 in 9dc0727

.map(unsplitList -> new TokenTextSplitter().apply(unsplitList));

Hey Craig, I'm trying to figure out how to apply this to my json api doc. This is my first time using the TokenTextSplitter. I guess I'm worried that it is sort of randomly splitting the text by length when the json has some very clear structure that would probably make more sense for splitting.

Can you point me in the right direction?

@Component
public class ReferenceDocsLoader {

    private static final Logger log = LoggerFactory.getLogger(ReferenceDocsLoader.class);
    private final JdbcClient jdbcClient;
    private final VectorStore vectorStore;
    @Value("classpath:/docs/api-docs.json")
    private Resource apiDocs;

    public ReferenceDocsLoader(JdbcClient jdbcClient, VectorStore vectorStore) {
        this.jdbcClient = jdbcClient;
        this.vectorStore = vectorStore;
    }

    @PostConstruct
    public void init() {
        Integer count = jdbcClient.sql("select count(*) from vector_store")
                .query(Integer.class)
                .single();

        log.info("Current count of the Vector Store: {}", count);
        if (count == 0) {
            try {
                String json = Files.readString(Paths.get(apiDocs.getURI()));
                Document doc = new Document(json);
                List<Document> subDocs = new TokenTextSplitter().apply(List.of(doc));
                vectorStore.accept(subDocs);
            } catch (IOException e) {
                log.error(e.getMessage(), e);
            }
        }
    }
}