Google Cloud boosts AlloyDB search for CJK text with Gemini

Google Cloud has outlined how to use AlloyDB AI Functions with Gemini models to improve search indexing for Chinese, Japanese and Korean text in AlloyDB. The approach targets full-text and hybrid search workloads in managed PostgreSQL environments.

The update focuses on a long-standing problem in database search. Standard PostgreSQL text parsing usually assumes words are separated by spaces, which creates difficulties for logographic languages written as continuous text, with punctuation providing most visible boundaries.

In those cases, a database may treat an entire sentence as a single lexeme instead of splitting it into searchable terms. As a result, a user searching for a specific phrase or noun within a longer Chinese sentence may not retrieve a result unless the full sentence is entered exactly.

AlloyDB can now call Gemini models directly from SQL through in-database AI Functions, allowing developers to apply word segmentation and stop-word removal before creating full-text indexes. Google Cloud presented the model-led approach as an alternative to third-party PostgreSQL extensions and external preprocessing services.

Those older methods can be difficult to use in managed database services. Extensions such as Chinese tokenisers may not be supported, while external preprocessing adds data movement, application logic and operational overhead.

In-database flow

The proposed design keeps raw text, segmented text, search vectors and embeddings in the same table. In Google Cloud's example, generated columns ensure that when segmented text is updated, AlloyDB automatically refreshes both the full-text search vector and the vector embedding.

For large document collections, Google Cloud described a batching process built with a PL/pgSQL stored procedure. Instead of sending one model request per row, the procedure groups rows into arrays, submits a batched call to Gemini, unpacks the returned outputs with GENERATE_SERIES, updates the target records and commits each batch immediately.

This method is intended to reduce row-locking issues and rollback risks in long-running update jobs. It also aims to avoid alignment problems that can occur when parallel cursor streams drift out of step.

One example in the documentation uses the Chinese sentence "你们研究所有十个图书馆". Without segmentation, PostgreSQL may index the full string as a single unit. After preprocessing with Gemini, the sentence is stored with spaces between meaningful words, making terms such as "研究所" and "图书馆" searchable.

Search options

Google Cloud also outlined how text search configuration should vary by dataset. If a database contains only Chinese text, developers may prefer PostgreSQL's simple configuration because it leaves the segmented output largely untouched apart from lowercasing.

For bilingual datasets that mix Chinese with English product names, technical terms or brand references, the english configuration may be more practical. In that setup, English words can still be normalised and common English stop words removed, while Chinese characters remain intact as exact lexemes.

The same preprocessing logic can also be applied at query time. A user search can be passed to Gemini from SQL, with instructions to segment the text and remove low-value grammatical words before the database converts the output into a text search query.

RUM and vectors

For ranking, Google Cloud pointed to AlloyDB's support for RUM indexes on search vectors. Unlike a standard GIN index, a RUM index stores lexeme positions directly, allowing the database to calculate relevance and word distance inside the index.

In the company's example, a query converts a segmented search string into a PostgreSQL tsquery, filters matching rows and sorts them by distance. Google Cloud said this can return relevant matches in milliseconds.

It also described a hybrid search pattern that combines full-text search with vector search. In that design, a ScaNN index is created on an embedding column generated from the multilingual Gemini embedding model, and the results from text and vector retrieval are merged with Reciprocal Rank Fusion.

The SQL pattern uses separate common table expressions for vector and text retrieval, then combines the ranked lists with a full outer join and calculates a final score from each document's ranking in the two result sets. The goal is to surface both exact keyword matches and semantically related documents.

The broader message is that more AI processing can now happen inside the database rather than in external application layers. That could appeal to organisations seeking multilingual search and vector retrieval without maintaining separate search stacks or moving data out of transactional systems.

Developers can build bilingual and hybrid search workflows in AlloyDB using standard SQL and stored procedures instead of external microservices, according to Google Cloud.