Reading database configuration parameters stored in Google Firestore — Configuration parameters are settings, that allows an application to determine essential values to function. Determination for configuration store and its management depends on various factors including the environment, deployment methodologies and type of configuration elements. …

Dynamic SQL using Jinja templating language for Python — Creating hand-crafted SQL is cumbersome and time consuming. Therefore, with an effective automation process, we can overcome these challenges. Jinja template provides the attributes and parameters to dynamically generate SQL queries and greatly saves time and effort. …

A guided approach to connect federated data sources — Data Federation is a holistic approach adopted by organizations to harvest meaningful insights, by collecting information from disparate data sources to host a unified data control point. This enables organizations to implement a centralized repository for enabling analytics. A federated data source, known as an external data source, is a…

Effective utilization of Cloud Function using BigQuery computing power — Google Cloud Functions is a serverless execution environment designed for building and connecting cloud services. It enables fully managed infrastructure, so that cloud users need to focus on application code without worrying about underlying hardware and software. Use Case Design a data processing pipeline to load data from Cloud Storage to BigQuery

Managing fine-grained data access control for sensitive data — BigQuery provides fine-grained access to columns using policy tags. When a user tries to access a restricted column at query time, BigQuery checks for column policy tag and its policy to see if the user is authorized to access the data. Following steps outlines implementation strategy for managing data access…

Systematic approach to generate data quality statistics from BigQuery tables — Data quality analysis enables organizations to reduce the cost of identifying and fixing bad data in the systems. It is an integral part of an organization to harvest meaningful insights. In this Blog, few important data quality statistics gathered, which can ensure only trusted data is used for decision-making, which…

Password Encryption at Rest and in Transit — Problem Statement Securing sensitive information begins with a proper understanding of security controls and the protection of passwords using modern encryption algorithms. Over the past decades, we have seen major security breaches exposing numerous usernames and passwords. As a result, millions of accounts impacted, and data compromised. With every password breach raises…

360-degree view of enterprise logs and point-in-time root cause analysis — Problem Statement IT infrastructure optimization is paramount to the success of any organization. However, organizations take a reactive approach to address this need after realizing cost overruns or bottlenecks in the IT infrastructure. The need of the hour is to have a proactive approach to accomplish desired goals. Enterprise logging considered as…

A solution to ingest real-time streaming data into BigQuery using serverless computing — Solution This solution is designed to ingest streaming data into BigQuery using Google cloud serverless platform without worrying about the underlying infrastructure. It also enables synchronous data read/write capabilities and contributes linear capacity planning & data accuracy. …