feat: Add GCP service account authentication support

README.md

@@ -67,6 +67,11 @@ This tap:
   - Process/send records to target
 
 ## Authentication
+
+This tap supports two authentication methods: OAuth2 (user credentials) and Service Account (for automated pipelines).
+
+### OAuth2 Authentication (User Credentials)
+
 The [**Google Sheets Setup & Authentication**](https://drive.google.com/open?id=1FojlvtLwS0-BzGS37R0jEXtwSHqSiO1Uw-7RKQQO-C4) Google Doc provides instructions show how to configure the Google Cloud API credentials to enable Google Drive and Google Sheets APIs, configure Google Cloud to authorize/verify your domain ownership, generate an API key (client_id, client_secret), authenticate and generate a refresh_token, and prepare your tap config.json with the necessary parameters.
 - Enable Google Sheets API and Authorization Scope: https://www.googleapis.com/auth/spreadsheets.readonly
 - Tap config.json parameters:
@@ -76,6 +81,70 @@ The [**Google Sheets Setup & Authentication**](https://drive.google.com/open?id=
   - spreadsheet_id: unique identifier for each spreadsheet in Google Drive
   - start_date: absolute minimum start date to check file modified
   - user_agent: tap-name and email address; identifies your application in the Remote API server logs
+  - sheet_names (optional): array of sheet names to extract; if omitted, all sheets are discovered
+
+### Service Account Authentication
+
+Service accounts are ideal for automated pipelines, CI/CD environments, and server-to-server integrations where user interaction is not possible.
+
+#### Setup Instructions
+
+1. **Create a Service Account**
+   - Go to [GCP Console > IAM & Admin > Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts)
+   - Click "Create Service Account"
+   - Give it a descriptive name (e.g., `tap-google-sheets-reader`)
+   - Click "Create and Continue", then "Done"
+
+2. **Enable Required APIs**
+   - Go to [APIs & Services > Library](https://console.cloud.google.com/apis/library)
+   - Enable "Google Sheets API"
+   - Enable "Google Drive API"
+
+3. **Create and Download JSON Key**
+   - In Service Accounts, click on your service account
+   - Go to "Keys" tab > "Add Key" > "Create new key"
+   - Select "JSON" format and click "Create"
+   - Save the downloaded key file securely
+
+4. **Share Spreadsheets with Service Account**
+   - Copy the service account email (e.g., `sa-name@project-id.iam.gserviceaccount.com`)
+   - Open your Google Spreadsheet
+   - Click "Share" and add the service account email with "Viewer" permission
+
+#### Configuration Examples
+
+**Using a credentials file path:**
+```json
+{
+    "credentials_file": "/path/to/service-account-key.json",
+    "spreadsheet_id": "YOUR_SPREADSHEET_ID",
+    "start_date": "2024-01-01T00:00:00Z",
+    "user_agent": "tap-google-sheets <sa@project.iam.gserviceaccount.com>"
+}
+```
+
+**Using inline JSON (for CI/CD environments):**
+```json
+{
+    "credentials_json": "{\"type\": \"service_account\", \"project_id\": \"...\", ...}",
+    "spreadsheet_id": "YOUR_SPREADSHEET_ID",
+    "start_date": "2024-01-01T00:00:00Z",
+    "user_agent": "tap-google-sheets <sa@project.iam.gserviceaccount.com>"
+}
+```
+
+#### Pro Tip: Google Groups for Permission Management
+
+Instead of sharing spreadsheets individually with service accounts, use Google Groups:
+
+1. Create a Google Group (e.g., `data-readers@yourdomain.com`)
+2. Add your service account email as a member of the group
+3. Share spreadsheets with the Google Group instead
+
+This approach makes it easier to:
+- Manage permissions across multiple service accounts
+- Grant access to new spreadsheets by sharing with one group
+- Audit who has access to your data
 
 ## Quick Start
 
@@ -115,6 +184,22 @@ The [**Google Sheets Setup & Authentication**](https://drive.google.com/open?id=
         "request_timeout": 300
     }
     ```
+
+    **Optional Configuration Parameters:**
+
+    - `sheet_names` (array of strings): Specify which sheets to extract from the spreadsheet. If provided, only these sheets will be discovered, and they will be automatically selected for extraction. If omitted, all sheets are discovered (but not automatically selected). This parameter works seamlessly with `meltano el` without requiring manual catalog editing.
+
+    Example with sheet filtering:
+    ```json
+    {
+        "spreadsheet_id": "YOUR_GOOGLE_SPREADSHEET_ID",
+        "sheet_names": ["Marco", "Sheet1", "Q4 Results"],
+        "start_date": "2019-01-01T00:00:00Z",
+        "user_agent": "tap-google-sheets <api_user_email@example.com>"
+    }
+    ```
+
+    **Note:** When `sheet_names` is specified, the filtered sheets are automatically selected in the catalog, making them immediately available for extraction without additional configuration.
 
     Optionally, also create a `state.json` file. `currently_syncing` is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.
     Only the `performance_reports` uses a bookmark. The date-time bookmark is stored in a nested structure based on the endpoint, site, and sub_type.The `request_timeout` is an optional paramater to set timeout for requests. Default: 300 seconds

config.json.example

@@ -1,9 +1,20 @@
 {
+    "_comment": "Option 1: OAuth2 Authentication (for user-delegated access)",
     "client_id": "YOUR_CLIENT_ID",
     "client_secret": "YOUR_CLIENT_SECRET",
     "refresh_token": "YOUR_REFRESH_TOKEN",
+
+    "_comment2": "Option 2: Service Account Authentication (for automated pipelines)",
+    "_comment3": "Use either credentials_file OR credentials_json, not both",
+    "credentials_file": "/path/to/service-account-key.json",
+    "credentials_json": "{\"type\": \"service_account\", \"project_id\": \"...\", ...}",
+
+    "_comment4": "Required fields for all authentication methods",
     "spreadsheet_id": "YOUR_GOOGLE_SPREADSHEET_ID",
     "start_date": "2019-01-01T00:00:00Z",
-    "user_agent": "tap-google-search-console <api_user_email@example.com>",
-    "request_timeout": 300
+    "user_agent": "tap-google-sheets <api_user_email@example.com>",
+    "request_timeout": 300,
+
+    "_comment5": "Optional: Filter to specific sheets (if omitted, all sheets are discovered)",
+    "sheet_names": ["Sheet1", "Sheet2", "Marco"]
 }

setup.py

@@ -11,7 +11,8 @@
       install_requires=[
           'backoff==2.2.1',
           'requests==2.33.0',
-          'singer-python==6.8.0'
+          'singer-python==6.8.0',
+          'google-auth>=2.0.0,<3.0.0'
       ],
       extras_require={
           'test': [

tap_google_sheets/__init__.py

@@ -11,19 +11,67 @@
 
 LOGGER = singer.get_logger()
 
+# Base required config keys (auth keys validated separately)
 REQUIRED_CONFIG_KEYS = [
-    'client_id',
-    'client_secret',
-    'refresh_token',
     'spreadsheet_id',
     'start_date',
     'user_agent'
 ]
 
-def do_discover(client, spreadsheet_id):
+
+def validate_auth_config(config):
+    """
+    Validate authentication configuration.
+
+    Either OAuth2 credentials (client_id, client_secret, refresh_token) OR
+    service account credentials (credentials_file OR credentials_json) must be provided,
+    but not both.
+
+    Returns:
+        str: 'oauth2' or 'service_account' indicating which auth type to use
+
+    Raises:
+        Exception: If config is invalid
+    """
+    # Check for OAuth2 credentials
+    has_oauth2 = all(key in config for key in ['client_id', 'client_secret', 'refresh_token'])
+
+    # Check for service account credentials
+    has_credentials_file = 'credentials_file' in config and config['credentials_file']
+    has_credentials_json = 'credentials_json' in config and config['credentials_json']
+    has_service_account = has_credentials_file or has_credentials_json
+
+    # Validate mutual exclusivity
+    if has_oauth2 and has_service_account:
+        raise Exception(
+            'Invalid config: Cannot specify both OAuth2 credentials (client_id, client_secret, '
+            'refresh_token) and service account credentials (credentials_file or credentials_json). '
+            'Please use one authentication method only.'
+        )
+
+    if not has_oauth2 and not has_service_account:
+        raise Exception(
+            'Invalid config: Must specify either OAuth2 credentials (client_id, client_secret, '
+            'refresh_token) OR service account credentials (credentials_file or credentials_json).'
+        )
+
+    if has_credentials_file and has_credentials_json:
+        raise Exception(
+            'Invalid config: Cannot specify both credentials_file and credentials_json. '
+            'Please use one service account credential source only.'
+        )
+
+    if has_service_account:
+        LOGGER.info('Using service account authentication')
+        return 'service_account'
+    else:
+        LOGGER.info('Using OAuth2 authentication')
+        return 'oauth2'
+
+def do_discover(client, config):
 
     LOGGER.info('Starting discover')
-    catalog = discover(client, spreadsheet_id)
+    catalog = discover(client, config)
     json.dump(catalog.to_dict(), sys.stdout, indent=2)
     LOGGER.info('Finished discover')
 
@@ -32,27 +80,39 @@ def do_discover(client, spreadsheet_id):
 def main():
 
     parsed_args = singer.utils.parse_args(REQUIRED_CONFIG_KEYS)
+    config = parsed_args.config
+
+    # Validate authentication config
+    auth_type = validate_auth_config(config)
+
+    # Build GoogleClient kwargs based on auth type
+    client_kwargs = {
+        'request_timeout': config.get('request_timeout'),
+        'user_agent': config['user_agent']
+    }
+
+    if auth_type == 'oauth2':
+        client_kwargs['client_id'] = config['client_id']
+        client_kwargs['client_secret'] = config['client_secret']
+        client_kwargs['refresh_token'] = config['refresh_token']
+    else:  # service_account
+        client_kwargs['credentials_file'] = config.get('credentials_file')
+        client_kwargs['credentials_json'] = config.get('credentials_json')
 
-    with GoogleClient(parsed_args.config['client_id'],
-                      parsed_args.config['client_secret'],
-                      parsed_args.config['refresh_token'],
-                      parsed_args.config.get('request_timeout'),
-                      parsed_args.config['user_agent']
-                      ) as client:
+    with GoogleClient(**client_kwargs) as client:
 
         state = {}
         if parsed_args.state:
             state = parsed_args.state
 
-        config = parsed_args.config
         spreadsheet_id = config.get('spreadsheet_id')
 
         if parsed_args.discover:
-            do_discover(client, spreadsheet_id)
+            do_discover(client, config)
         else:
             sync(client=client,
                  config=config,
-                 catalog=parsed_args.catalog or discover(client, spreadsheet_id),
+                 catalog=parsed_args.catalog or discover(client, config),
                  state=state)
 
 if __name__ == '__main__':

tap_google_sheets/client.py

@@ -1,17 +1,27 @@
 from datetime import datetime, timedelta
 from collections import OrderedDict
+import json
 import backoff
 import requests
 import singer
 from singer import metrics
 from singer import utils
 from requests.exceptions import Timeout, ConnectionError
 
+from google.oauth2 import service_account
+from google.auth.transport.requests import Request as GoogleAuthRequest
+
 BASE_URL = 'https://www.googleapis.com'
 GOOGLE_TOKEN_URI = 'https://oauth2.googleapis.com/token'
 LOGGER = singer.get_logger()
 REQUEST_TIMEOUT = 300
 
+# API scopes required for Google Sheets and Drive access
+SCOPES = [
+    'https://www.googleapis.com/auth/spreadsheets.readonly',
+    'https://www.googleapis.com/auth/drive.readonly'
+]
+
 class Server5xxError(Exception):
     pass
 
@@ -132,14 +142,29 @@ def raise_for_error(response):
 
 class GoogleClient: # pylint: disable=too-many-instance-attributes
     def __init__(self,
-                 client_id,
-                 client_secret,
-                 refresh_token,
+                 client_id=None,
+                 client_secret=None,
+                 refresh_token=None,
                  request_timeout=REQUEST_TIMEOUT,
-                 user_agent=None):
+                 user_agent=None,
+                 credentials_file=None,
+                 credentials_json=None):
+        # OAuth2 credentials
         self.__client_id = client_id
         self.__client_secret = client_secret
         self.__refresh_token = refresh_token
+
+        # Service account credentials
+        self.__credentials_file = credentials_file
+        self.__credentials_json = credentials_json
+        self.__sa_credentials = None
+
+        # Determine auth type
+        if credentials_file or credentials_json:
+            self.__auth_type = 'service_account'
+        else:
+            self.__auth_type = 'oauth2'
+
         self.__user_agent = user_agent
         self.__access_token = None
         self.__expires = None
@@ -166,11 +191,33 @@ def __enter__(self):
     def __exit__(self, exception_type, exception_value, traceback):
         self.__session.close()
 
+    def _load_service_account_credentials(self):
+        """Load and return service account credentials from file or JSON string."""
+        if self.__sa_credentials is not None:
+            return self.__sa_credentials
+
+        if self.__credentials_file:
+            LOGGER.info('Loading service account credentials from file: {}'.format(self.__credentials_file))
+            self.__sa_credentials = service_account.Credentials.from_service_account_file(
+                self.__credentials_file,
+                scopes=SCOPES
+            )
+        elif self.__credentials_json:
+            LOGGER.info('Loading service account credentials from inline JSON')
+            credentials_info = json.loads(self.__credentials_json)
+            self.__sa_credentials = service_account.Credentials.from_service_account_info(
+                credentials_info,
+                scopes=SCOPES
+            )
+
+        return self.__sa_credentials
+
     @backoff.on_exception(backoff.expo,
                           Server5xxError,
                           max_tries=5,
                           factor=2)
-    def get_access_token(self):
+    def _get_oauth2_token(self):
+        """Get access token using OAuth2 refresh token flow."""
         # The refresh_token never expires and may be used many times to generate each access_token
         # Since the refresh_token does not expire, it is not included in get access_token response
         if self.__access_token is not None and self.__expires > datetime.utcnow():
@@ -200,7 +247,38 @@ def get_access_token(self):
         data = response.json()
         self.__access_token = data['access_token']
         self.__expires = datetime.utcnow() + timedelta(seconds=data['expires_in'])
-        LOGGER.info('Authorized, token expires = {}'.format(self.__expires))
+        LOGGER.info('Authorized via OAuth2, token expires = {}'.format(self.__expires))
+
+    @backoff.on_exception(backoff.expo,
+                          Server5xxError,
+                          max_tries=5,
+                          factor=2)
+    def _get_service_account_token(self):
+        """Get access token using service account credentials."""
+        credentials = self._load_service_account_credentials()
+
+        # Check if token needs refresh (expired or not yet obtained)
+        if credentials.token is None or (credentials.expiry and credentials.expiry <= datetime.utcnow()):
+            LOGGER.info('Refreshing service account token')
+            credentials.refresh(GoogleAuthRequest())
+
+        self.__access_token = credentials.token
+        self.__expires = credentials.expiry
+        LOGGER.info('Authorized via service account, token expires = {}'.format(self.__expires))
+
+    @backoff.on_exception(backoff.expo,
+                          Server5xxError,
+                          max_tries=5,
+                          factor=2)
+    def get_access_token(self):
+        """Get access token using appropriate authentication method."""
+        if self.__access_token is not None and self.__expires and self.__expires > datetime.utcnow():
+            return
+
+        if self.__auth_type == 'service_account':
+            self._get_service_account_token()
+        else:
+            self._get_oauth2_token()
 
 
     # Backoff request for 5 times at an interval of 10 seconds when we get Timeout error