Zero-shot predictions in Pandas using the new BigQuery Accessor

Imagine running advanced machine learning workflows—from forecasting and classification to high-dimensional embedding generation—without ever leaving your familiar pandas DataFrame. With the new .bigquery accessor from BigQuery DataFrames (BigFrames), that vision is now a reality. You can seamlessly invoke BigQuery's enterprise-grade AI functions directly within pandas, scaling your data science and ML pipelines on cloud infrastructure . Take linear regression as an example. Traditionally, a pandas user who needs to run regression would have to extract data locally, manually select features, and train a custom scikit-learn model like this: code_block <ListValue: [StructValue([('code', "import pandas as pd\r\nfrom sklearn.linear_model import LinearRegression\r\nfrom sklearn.preprocessing import StandardScaler\r\nfrom sklearn.pipeline import make_pipeline\r\n\r\n# Load historical training data from a CSV\r\ntrain_df = pd.read_csv('historical_data.csv')\r\n\r\n# Separate features (X) and target (y)\r\nX_train = train_df[['feature_1', 'feature_2']]\r\ny_train = train_df['target_metric']\r\n\r\n# Build and train a modeling pipeline\r\n# The StandardScaler ensures features with different scales don't bias the model\r\nmodel_pipeline = make_pipeline(StandardScaler(), LinearRegression())\r\nmodel_pipeline.fit(X_train, y_train)\r\n\r\n# Load new data for inference from another CSV\r\nnew_data_df = pd.read_csv('new_data.csv')\r\nX_new = new_data_df[['feature_1', 'feature_2']]\r\n\r\n# Make predictions \r\n# (The pipeline automatically scales the new data before predicting)\r\nnew_data_df['predictions'] = model_pipeline.predict(X_new)\r\n\r\nprint(new_data_df.head())"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa7d69a4a90>)])]> Now, with the .bigquery accessor, you can bypass the local training process entirely and leverage powerful ai.predict() directly on your data in BigQuery: code_block <ListValue: [StructValue([('code', "import pandas as pd\r\nimport bigframes.pandas as bpd # Enables bigquery accessor\r\n\r\n# Set a GCP project to use AI functions.\r\nbpd.options.bigquery.project = 'my-project' \r\n\r\ntrain_df = pd.read_csv('historical_data.csv')\r\nnew_data_df = pd.read_csv('new_data.csv')\r\n\r\n# Make predictions with TabFM\r\npredictions = train_df.bigquery.ai.predict(new_data_df, label_col='target_metric')\r\nprint(predictions)"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa7d5ec1d90>)])]> As you can see, the ai.predict() is able to perform regression tasks directly over two pandas dataframes with a single function call, which greatly simplifies the entire ML process. How the BigQuery accessor works Under the hood, the .bigquery accessor uses BigQuery DataFrames library to transparently push your computations to Google Cloud. When you call an AI method like .ai.predict(), it doesn't compute the regression locally. Instead, it translates the pandas operation into Google Standard SQL, executes the corresponding foundational model (like TabFM) natively on BigQuery's infrastructure, and seamlessly returns the result as a standard pandas DataFrame. This allows you to easily process local datasets with powerful offerings unique to BigQuery. The following sections detail additional executable code examples. You can also find them all in this notebook. Environment setup Before executing the code samples in this post, ensure you have a Google Cloud project with the following roles granted (see more at permissions for AI functions): Run query jobs: BigQuery Job User (roles/bigquery.jobUser) Access a foundation model in Gemini Enterprise Agent Platform: Agent Platform User (roles/aiplatform.user) Next, install pandas and bigframes in your Jupyter environment. To enable the AI accessors, simply import the bigframes library and specify your Google Cloud project ID. This ensures all subsequent AI function calls are routed through your authorized project: code_block <ListValue: [StructValue([('code', 'import bigframes.pandas as bpd \r\n\r\n# Replace with your project ID \r\nbpd.options.bigquery.project = "my-project"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fa7d5ec2050>)])]> Once configured, you can call BigQuery AI functions directly on standard pandas objects. Functions on pandas series AI functions attached to a pandas `Series` accept single-column input sequences. Example: AI.EMBED Generating vector embeddings is a fundamental step in many modern NLP and semantic search pipelines. By using the `ai.embed()` function, you can compute embeddings for an entire column of text in a single, clean expression. code_block <ListValue: [StructValue([('code', "import pandas as pd\r\n\r\nanimals = pd.Series(['dog', 'fish'])\r\nanimals.bigquery.ai.embed(endpoint='text-embedding-005')\r\n# 0 {'result': array([ 1.78243860e-03, -1.10658340...\r\n# 1 {'result': array([-7.29714287e-03, 1.04725976.

Extras afișat conform termenilor RSS ai publicației · 112 cuvinte · textul integral și figurile sunt pe site-ul publicației