Data at Google Cloud Next '24!
Answer this question in the comments to earn a special badge: In your journey, what's one overlooked data skil...
Data Analytics
Want more from your data? Have questions about products like BigQuery, Dataflow, or Pub/Sub? You're in the right place. Join conversations around data warehouse modernization, streaming analytics, business intelligence, and more.
Focused on Looker? Engage with the Looker Community here.
Focused on Looker? Engage with the Looker Community here.
Forum Posts
Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
How to launch a Dataflow job with Cloud Composer?
I have a requirement to orchestrate a dataflow python job using cloud composer DAG.Did anyone came across this...
CSV data import using Dataflow
Hi, I am importing data from csv (file in storage) to BigQuery using dataflow. I am creating dataflow using ex...
Getting java.lang.NullPointerException from a working dataflow Job
Hi,I am getting the following error from a working dataflow job It keeps popping up even though the dataflow j...
Big Query EXPORT DATA to Cloud Storage - special characters write badly
I need to export data of a BigQuery table into CSV on Google Cloud Storage.I used the following:EXPORT DATA OP...
Does Datastream require Dataflow to process the data?
I am trying to understand Datastream's pricing for real-time analytics. It explains the $/GB for both initial ...
Unable to send SMTP Dag failure notifications
Hi, I need to send failure notifications from composer DAGs. I have created an app password for gmail smtp. Fo...
Getting java.lang.NullPointerException from a working dataflow Job
Hi, I am getting the following error from a working dataflow job It keeps popping up even though the dataflow ...
Selecting query in console is changing indentation
For the past weeks, the BigQuery console is changing the indentation (kind of like compressing the whitespaces...
Migrating On Premise Apache Airflow workflow scripts to Cloud Composer
I have an on-premise environment running Airflow v2.2.0 and wish to migrate all the workflows in this instance...
Is there a way to apply labels to a BigQuery query issued from a Dataflow job?
We are running Apache Beam on Google CLoud Dataflow. One of our jobs SELECTs data from BigQuery and inserts ea...
Unusually high amount of data processed when processing 27 GB Big query table with a script
Hi all,I want to break up a main table by 'advertiser' into different tables to a different project in Big Que...
Cloud Composer environment creation Error
Hi I have en an error when I try to create a Cloud Composer Environment : Composer Backend timed out. Currentl...
log4j vulnerability (CVE-2021-44228) impact on DataFlow
Hello,I've read this bulletin about the vulnerability:https://cloud.google.com/log4j2-security-advisory?hl=enB...
spark log files are missing
newbieI am unable to debug my spark jobs. They fail with an OOM exception. The only tool I have to debug are l...
bigquery limitations for DML (merge upsert)
hi we are keen to know if there are any limits of DML on a streaming table, where records can be in hundreds o...
Load files from Storage to BigQuery
Hello I have log files in Storage directory and I want to load them in Bigquery. In Python I execute blobs_all...
Cloud Composer
One of my Airflow DAG Task is getting failed. To resolve it, I'm trying to create an image for the latest gith...
Can I use (in python) load_table_from_uri with file generations (cloud versioning)?
I want to move CSV file from Cloud Storage Bucket to BQ and I want to use load_table_from_uri in the same time...
Flex slots and bytes billed
Hey,I've been testing flex slots in my company to bring the costs of scheduled queries down and have managed t...
pandas df from Functions to BigQuery
I would like to write a pandas df into Bigquery using load_table_from_dataframe. But it throws me this error:G...
Getting total emails sent in a year
Assuming we have an entire year's worth of bigquery data, does anyone have a good data studio or SQL script te...
LDAP access to Google Analytics and Google Tag Manager
Is there a way to provide LDAP access to Google Analytics and Google Tag Manager to user groups?
Local testing for cloud composer
Hello, Currently, we use cloud composer to use apache airflow (1.10.14+composer).Before we deploy our code, we...
push back new metrics to google analytics
Hi AllI want to use GCP to load Google Analytics data, calculate and create a new dimension with GCP, and fina...
Twitter shares how they run their data architecture on Google Cloud Platform
Architecture Diagram Learn how Twitter's data engineers built their social media platform on their Data Center...
Cloud Logging and Bigquery
There are some Google Logs route that i set to send certain logs to BigqueryThis routes are in a project and s...
Trouble with Dataflow Pub/Sub to ElasticSearch Template
Hello, I've been trying to use the template called "pub/sub to ElasticSearch" for dataflow. Except for some un...
Is GCS a storage for Hbase in Dataproc
Where does the writes happen for hbase in a Dataproc cluster?Is it GCS or PD?
Working with Row level Security and Python
I am working with Google Cloud Platform BigQuery and I have created a project in order to apply row level secu...
Big Query UseStorageAPI returning 404 not found error randomly
When using "UseStorageAPI=true" in Big Query API, it is returning "404 not found" and "Unable to parse content...
Articles from Product Experts and Googlers
Turbocharge your SQL with the new DBT on Composer Blueprint
There's a ready to use blueprint for running DBT on Cloud Composer available with an operational dashboard.
Building and Deploying AI Agents with LangChain on Vertex AI
In this blog post, we’ll walk through how LangChain on Vertex AI helps developers simplify the complexities of deploying and managing your AI agents.
Simplify SAP System Copies with Google Cloud Disk Snapshots
Learn how Google Cloud simplifies and streamlines SAP system copies, recoveries, and refreshes through innovative snapshot technology, minimizing downtime and maximizing efficiency.