diff --git a/README.md b/README.md index 8d79e24..6c5aaa9 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,11 @@ This tap: - Process/send records to target ## Authentication + +This tap supports two authentication methods: OAuth2 (user credentials) and Service Account (for automated pipelines). + +### OAuth2 Authentication (User Credentials) + The [**Google Sheets Setup & Authentication**](https://drive.google.com/open?id=1FojlvtLwS0-BzGS37R0jEXtwSHqSiO1Uw-7RKQQO-C4) Google Doc provides instructions show how to configure the Google Cloud API credentials to enable Google Drive and Google Sheets APIs, configure Google Cloud to authorize/verify your domain ownership, generate an API key (client_id, client_secret), authenticate and generate a refresh_token, and prepare your tap config.json with the necessary parameters. - Enable Google Sheets API and Authorization Scope: https://www.googleapis.com/auth/spreadsheets.readonly - Tap config.json parameters: @@ -76,6 +81,70 @@ The [**Google Sheets Setup & Authentication**](https://drive.google.com/open?id= - spreadsheet_id: unique identifier for each spreadsheet in Google Drive - start_date: absolute minimum start date to check file modified - user_agent: tap-name and email address; identifies your application in the Remote API server logs + - sheet_names (optional): array of sheet names to extract; if omitted, all sheets are discovered + +### Service Account Authentication + +Service accounts are ideal for automated pipelines, CI/CD environments, and server-to-server integrations where user interaction is not possible. + +#### Setup Instructions + +1. **Create a Service Account** + - Go to [GCP Console > IAM & Admin > Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) + - Click "Create Service Account" + - Give it a descriptive name (e.g., `tap-google-sheets-reader`) + - Click "Create and Continue", then "Done" + +2. **Enable Required APIs** + - Go to [APIs & Services > Library](https://console.cloud.google.com/apis/library) + - Enable "Google Sheets API" + - Enable "Google Drive API" + +3. **Create and Download JSON Key** + - In Service Accounts, click on your service account + - Go to "Keys" tab > "Add Key" > "Create new key" + - Select "JSON" format and click "Create" + - Save the downloaded key file securely + +4. **Share Spreadsheets with Service Account** + - Copy the service account email (e.g., `sa-name@project-id.iam.gserviceaccount.com`) + - Open your Google Spreadsheet + - Click "Share" and add the service account email with "Viewer" permission + +#### Configuration Examples + +**Using a credentials file path:** +```json +{ + "credentials_file": "/path/to/service-account-key.json", + "spreadsheet_id": "YOUR_SPREADSHEET_ID", + "start_date": "2024-01-01T00:00:00Z", + "user_agent": "tap-google-sheets " +} +``` + +**Using inline JSON (for CI/CD environments):** +```json +{ + "credentials_json": "{\"type\": \"service_account\", \"project_id\": \"...\", ...}", + "spreadsheet_id": "YOUR_SPREADSHEET_ID", + "start_date": "2024-01-01T00:00:00Z", + "user_agent": "tap-google-sheets " +} +``` + +#### Pro Tip: Google Groups for Permission Management + +Instead of sharing spreadsheets individually with service accounts, use Google Groups: + +1. Create a Google Group (e.g., `data-readers@yourdomain.com`) +2. Add your service account email as a member of the group +3. Share spreadsheets with the Google Group instead + +This approach makes it easier to: +- Manage permissions across multiple service accounts +- Grant access to new spreadsheets by sharing with one group +- Audit who has access to your data ## Quick Start @@ -115,6 +184,22 @@ The [**Google Sheets Setup & Authentication**](https://drive.google.com/open?id= "request_timeout": 300 } ``` + + **Optional Configuration Parameters:** + + - `sheet_names` (array of strings): Specify which sheets to extract from the spreadsheet. If provided, only these sheets will be discovered, and they will be automatically selected for extraction. If omitted, all sheets are discovered (but not automatically selected). This parameter works seamlessly with `meltano el` without requiring manual catalog editing. + + Example with sheet filtering: + ```json + { + "spreadsheet_id": "YOUR_GOOGLE_SPREADSHEET_ID", + "sheet_names": ["Marco", "Sheet1", "Q4 Results"], + "start_date": "2019-01-01T00:00:00Z", + "user_agent": "tap-google-sheets " + } + ``` + + **Note:** When `sheet_names` is specified, the filtered sheets are automatically selected in the catalog, making them immediately available for extraction without additional configuration. Optionally, also create a `state.json` file. `currently_syncing` is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off. Only the `performance_reports` uses a bookmark. The date-time bookmark is stored in a nested structure based on the endpoint, site, and sub_type.The `request_timeout` is an optional paramater to set timeout for requests. Default: 300 seconds diff --git a/config.json.example b/config.json.example index 5c07bce..264d20c 100644 --- a/config.json.example +++ b/config.json.example @@ -1,9 +1,20 @@ { + "_comment": "Option 1: OAuth2 Authentication (for user-delegated access)", "client_id": "YOUR_CLIENT_ID", "client_secret": "YOUR_CLIENT_SECRET", "refresh_token": "YOUR_REFRESH_TOKEN", + + "_comment2": "Option 2: Service Account Authentication (for automated pipelines)", + "_comment3": "Use either credentials_file OR credentials_json, not both", + "credentials_file": "/path/to/service-account-key.json", + "credentials_json": "{\"type\": \"service_account\", \"project_id\": \"...\", ...}", + + "_comment4": "Required fields for all authentication methods", "spreadsheet_id": "YOUR_GOOGLE_SPREADSHEET_ID", "start_date": "2019-01-01T00:00:00Z", - "user_agent": "tap-google-search-console ", - "request_timeout": 300 + "user_agent": "tap-google-sheets ", + "request_timeout": 300, + + "_comment5": "Optional: Filter to specific sheets (if omitted, all sheets are discovered)", + "sheet_names": ["Sheet1", "Sheet2", "Marco"] } diff --git a/setup.py b/setup.py index a2977b6..ac1a064 100644 --- a/setup.py +++ b/setup.py @@ -11,7 +11,8 @@ install_requires=[ 'backoff==2.2.1', 'requests==2.33.0', - 'singer-python==6.8.0' + 'singer-python==6.8.0', + 'google-auth>=2.0.0,<3.0.0' ], extras_require={ 'test': [ diff --git a/tap_google_sheets/__init__.py b/tap_google_sheets/__init__.py index 535f7c5..c2b782d 100644 --- a/tap_google_sheets/__init__.py +++ b/tap_google_sheets/__init__.py @@ -11,19 +11,67 @@ LOGGER = singer.get_logger() +# Base required config keys (auth keys validated separately) REQUIRED_CONFIG_KEYS = [ - 'client_id', - 'client_secret', - 'refresh_token', 'spreadsheet_id', 'start_date', 'user_agent' ] -def do_discover(client, spreadsheet_id): + +def validate_auth_config(config): + """ + Validate authentication configuration. + + Either OAuth2 credentials (client_id, client_secret, refresh_token) OR + service account credentials (credentials_file OR credentials_json) must be provided, + but not both. + + Returns: + str: 'oauth2' or 'service_account' indicating which auth type to use + + Raises: + Exception: If config is invalid + """ + # Check for OAuth2 credentials + has_oauth2 = all(key in config for key in ['client_id', 'client_secret', 'refresh_token']) + + # Check for service account credentials + has_credentials_file = 'credentials_file' in config and config['credentials_file'] + has_credentials_json = 'credentials_json' in config and config['credentials_json'] + has_service_account = has_credentials_file or has_credentials_json + + # Validate mutual exclusivity + if has_oauth2 and has_service_account: + raise Exception( + 'Invalid config: Cannot specify both OAuth2 credentials (client_id, client_secret, ' + 'refresh_token) and service account credentials (credentials_file or credentials_json). ' + 'Please use one authentication method only.' + ) + + if not has_oauth2 and not has_service_account: + raise Exception( + 'Invalid config: Must specify either OAuth2 credentials (client_id, client_secret, ' + 'refresh_token) OR service account credentials (credentials_file or credentials_json).' + ) + + if has_credentials_file and has_credentials_json: + raise Exception( + 'Invalid config: Cannot specify both credentials_file and credentials_json. ' + 'Please use one service account credential source only.' + ) + + if has_service_account: + LOGGER.info('Using service account authentication') + return 'service_account' + else: + LOGGER.info('Using OAuth2 authentication') + return 'oauth2' + +def do_discover(client, config): LOGGER.info('Starting discover') - catalog = discover(client, spreadsheet_id) + catalog = discover(client, config) json.dump(catalog.to_dict(), sys.stdout, indent=2) LOGGER.info('Finished discover') @@ -32,27 +80,39 @@ def do_discover(client, spreadsheet_id): def main(): parsed_args = singer.utils.parse_args(REQUIRED_CONFIG_KEYS) + config = parsed_args.config + + # Validate authentication config + auth_type = validate_auth_config(config) + + # Build GoogleClient kwargs based on auth type + client_kwargs = { + 'request_timeout': config.get('request_timeout'), + 'user_agent': config['user_agent'] + } + + if auth_type == 'oauth2': + client_kwargs['client_id'] = config['client_id'] + client_kwargs['client_secret'] = config['client_secret'] + client_kwargs['refresh_token'] = config['refresh_token'] + else: # service_account + client_kwargs['credentials_file'] = config.get('credentials_file') + client_kwargs['credentials_json'] = config.get('credentials_json') - with GoogleClient(parsed_args.config['client_id'], - parsed_args.config['client_secret'], - parsed_args.config['refresh_token'], - parsed_args.config.get('request_timeout'), - parsed_args.config['user_agent'] - ) as client: + with GoogleClient(**client_kwargs) as client: state = {} if parsed_args.state: state = parsed_args.state - config = parsed_args.config spreadsheet_id = config.get('spreadsheet_id') if parsed_args.discover: - do_discover(client, spreadsheet_id) + do_discover(client, config) else: sync(client=client, config=config, - catalog=parsed_args.catalog or discover(client, spreadsheet_id), + catalog=parsed_args.catalog or discover(client, config), state=state) if __name__ == '__main__': diff --git a/tap_google_sheets/client.py b/tap_google_sheets/client.py index c187a5e..fd45a2f 100644 --- a/tap_google_sheets/client.py +++ b/tap_google_sheets/client.py @@ -1,5 +1,6 @@ from datetime import datetime, timedelta from collections import OrderedDict +import json import backoff import requests import singer @@ -7,11 +8,20 @@ from singer import utils from requests.exceptions import Timeout, ConnectionError +from google.oauth2 import service_account +from google.auth.transport.requests import Request as GoogleAuthRequest + BASE_URL = 'https://www.googleapis.com' GOOGLE_TOKEN_URI = 'https://oauth2.googleapis.com/token' LOGGER = singer.get_logger() REQUEST_TIMEOUT = 300 +# API scopes required for Google Sheets and Drive access +SCOPES = [ + 'https://www.googleapis.com/auth/spreadsheets.readonly', + 'https://www.googleapis.com/auth/drive.readonly' +] + class Server5xxError(Exception): pass @@ -132,14 +142,29 @@ def raise_for_error(response): class GoogleClient: # pylint: disable=too-many-instance-attributes def __init__(self, - client_id, - client_secret, - refresh_token, + client_id=None, + client_secret=None, + refresh_token=None, request_timeout=REQUEST_TIMEOUT, - user_agent=None): + user_agent=None, + credentials_file=None, + credentials_json=None): + # OAuth2 credentials self.__client_id = client_id self.__client_secret = client_secret self.__refresh_token = refresh_token + + # Service account credentials + self.__credentials_file = credentials_file + self.__credentials_json = credentials_json + self.__sa_credentials = None + + # Determine auth type + if credentials_file or credentials_json: + self.__auth_type = 'service_account' + else: + self.__auth_type = 'oauth2' + self.__user_agent = user_agent self.__access_token = None self.__expires = None @@ -166,11 +191,33 @@ def __enter__(self): def __exit__(self, exception_type, exception_value, traceback): self.__session.close() + def _load_service_account_credentials(self): + """Load and return service account credentials from file or JSON string.""" + if self.__sa_credentials is not None: + return self.__sa_credentials + + if self.__credentials_file: + LOGGER.info('Loading service account credentials from file: {}'.format(self.__credentials_file)) + self.__sa_credentials = service_account.Credentials.from_service_account_file( + self.__credentials_file, + scopes=SCOPES + ) + elif self.__credentials_json: + LOGGER.info('Loading service account credentials from inline JSON') + credentials_info = json.loads(self.__credentials_json) + self.__sa_credentials = service_account.Credentials.from_service_account_info( + credentials_info, + scopes=SCOPES + ) + + return self.__sa_credentials + @backoff.on_exception(backoff.expo, Server5xxError, max_tries=5, factor=2) - def get_access_token(self): + def _get_oauth2_token(self): + """Get access token using OAuth2 refresh token flow.""" # The refresh_token never expires and may be used many times to generate each access_token # Since the refresh_token does not expire, it is not included in get access_token response if self.__access_token is not None and self.__expires > datetime.utcnow(): @@ -200,7 +247,38 @@ def get_access_token(self): data = response.json() self.__access_token = data['access_token'] self.__expires = datetime.utcnow() + timedelta(seconds=data['expires_in']) - LOGGER.info('Authorized, token expires = {}'.format(self.__expires)) + LOGGER.info('Authorized via OAuth2, token expires = {}'.format(self.__expires)) + + @backoff.on_exception(backoff.expo, + Server5xxError, + max_tries=5, + factor=2) + def _get_service_account_token(self): + """Get access token using service account credentials.""" + credentials = self._load_service_account_credentials() + + # Check if token needs refresh (expired or not yet obtained) + if credentials.token is None or (credentials.expiry and credentials.expiry <= datetime.utcnow()): + LOGGER.info('Refreshing service account token') + credentials.refresh(GoogleAuthRequest()) + + self.__access_token = credentials.token + self.__expires = credentials.expiry + LOGGER.info('Authorized via service account, token expires = {}'.format(self.__expires)) + + @backoff.on_exception(backoff.expo, + Server5xxError, + max_tries=5, + factor=2) + def get_access_token(self): + """Get access token using appropriate authentication method.""" + if self.__access_token is not None and self.__expires and self.__expires > datetime.utcnow(): + return + + if self.__auth_type == 'service_account': + self._get_service_account_token() + else: + self._get_oauth2_token() # Backoff request for 5 times at an interval of 10 seconds when we get Timeout error diff --git a/tap_google_sheets/discover.py b/tap_google_sheets/discover.py index 1440dee..3b15516 100644 --- a/tap_google_sheets/discover.py +++ b/tap_google_sheets/discover.py @@ -1,13 +1,25 @@ from singer.catalog import Catalog, CatalogEntry, Schema from tap_google_sheets.schema import STREAMS +import singer +from singer import metadata +LOGGER = singer.get_logger() -def discover(client, spreadsheet_id): + +def discover(client, config): catalog = Catalog([]) + spreadsheet_id = config.get('spreadsheet_id') + sheet_names = config.get('sheet_names', []) + + # Log sheet filtering info + if sheet_names: + LOGGER.info('Filtering sheets to: %s', sheet_names) + else: + LOGGER.info('No sheet_names filter specified, discovering all sheets') for stream, stream_obj in STREAMS.items(): stream_object = stream_obj(client, spreadsheet_id) - schemas, field_metadata = stream_object.get_schemas() + schemas, field_metadata = stream_object.get_schemas(sheet_names_filter=sheet_names) # loop over the schema and prepare catalog for stream_name, schema_dict in schemas.items(): @@ -15,6 +27,18 @@ def discover(client, spreadsheet_id): schema = Schema.from_dict(schema_dict) mdata = field_metadata[stream_name] + # If sheet_names filter is specified, automatically select matching sheets + # This makes meltano el work without needing manual catalog selection + if sheet_names and len(sheet_names) > 0: + # Check if this stream is a sheet (not metadata streams) + if stream_name not in ['spreadsheet_metadata', 'sheet_metadata', 'sheets_loaded']: + # Mark sheet as selected if it's in the filter + if stream_name in sheet_names: + mdata_map = metadata.to_map(mdata) + mdata = metadata.write(mdata_map, (), 'selected', True) + mdata = metadata.to_list(mdata_map) + LOGGER.info('Auto-selecting sheet (in sheet_names filter): %s', stream_name) + # get the primary keys for the stream # if the stream is from STREAM, then get the key_properties # else use the "table-key-properties" from the metadata diff --git a/tap_google_sheets/schema.py b/tap_google_sheets/schema.py index c7f79a1..0721b71 100644 --- a/tap_google_sheets/schema.py +++ b/tap_google_sheets/schema.py @@ -249,8 +249,8 @@ def get_sheet_metadata(sheet, spreadsheet_id, client): try: sheet_json_schema, columns = get_sheet_schema_columns(sheet_metadata) except Exception as err: - LOGGER.warning('{}'.format(err)) - LOGGER.warning('SKIPPING Malformed sheet: {}'.format(sheet_title)) - sheet_json_schema, columns = None, None + LOGGER.error('{}'.format(err)) + LOGGER.error('SKIPPING Malformed sheet: {}'.format(sheet_title)) + raise return sheet_json_schema, columns diff --git a/tap_google_sheets/streams.py b/tap_google_sheets/streams.py index ac95ca7..78b220d 100644 --- a/tap_google_sheets/streams.py +++ b/tap_google_sheets/streams.py @@ -144,7 +144,7 @@ def get_path(self, sheet_title_encoded=""): # return path and query string return path, querystring - def get_schemas(self): + def get_schemas(self, sheet_names_filter=None): """ return schema for streams """ @@ -253,12 +253,12 @@ class SpreadSheetMetadata(GoogleSheets): "includeGridData": "false" } - def get_schemas(self): + def get_schemas(self, sheet_names_filter=None): """ Get schema for spreadsheet and generate schema for the sheets in the spreadsheet """ # get schema of spreadsheet metadata - schemas, field_metadata = super().get_schemas() + schemas, field_metadata = super().get_schemas(sheet_names_filter=sheet_names_filter) # prepare schema for sheets in the spreadsheet api = self.api @@ -271,12 +271,19 @@ def get_schemas(self): if sheets: # Loop thru each worksheet in spreadsheet for sheet in sheets: + sheet_title = sheet.get('properties', {}).get('title') + + # Apply sheet_names filter if provided + if sheet_names_filter and len(sheet_names_filter) > 0: + if sheet_title not in sheet_names_filter: + LOGGER.info('Skipping sheet (not in sheet_names filter): %s', sheet_title) + continue + # GET sheet_json_schema for each worksheet (from function above) sheet_json_schema, columns = schema.get_sheet_metadata(sheet, self.spreadsheet_id, self.client) # SKIP empty sheets (where sheet_json_schema and columns are None) if sheet_json_schema and columns: - sheet_title = sheet.get('properties', {}).get('title') schemas[sheet_title] = sheet_json_schema sheet_mdata = metadata.new() sheet_mdata = metadata.get_standard_metadata( @@ -408,7 +415,7 @@ class SheetsLoadData(GoogleSheets): replication_method = "FULL_TABLE" params = {} - def load_data(self, catalog, state, selected_streams, sheets, spreadsheet_time_extracted): + def load_data(self, catalog, state, selected_streams, sheets, spreadsheet_time_extracted, sheet_names_filter=None): """ Load sheet's records if that sheet is selected for sync """ @@ -421,6 +428,12 @@ def load_data(self, catalog, state, selected_streams, sheets, spreadsheet_time_e sheet_title = sheet.get('properties', {}).get('title') sheet_id = sheet.get('properties', {}).get('sheetId') + # Apply sheet_names filter if provided + if sheet_names_filter and len(sheet_names_filter) > 0: + if sheet_title not in sheet_names_filter: + LOGGER.info('Skipping sheet (not in sheet_names filter): %s', sheet_title) + continue + # GET sheet_metadata and columns sheet_schema, columns = schema.get_sheet_metadata(sheet, self.spreadsheet_id, self.client) # LOGGER.info('sheet_schema: {}'.format(sheet_schema)) diff --git a/tap_google_sheets/sync.py b/tap_google_sheets/sync.py index e518140..2ec768c 100644 --- a/tap_google_sheets/sync.py +++ b/tap_google_sheets/sync.py @@ -45,12 +45,16 @@ def sync(client, config, catalog, state): # class to load sheet's data sheets_load_data = SheetsLoadData(client, config.get("spreadsheet_id"), config.get("start_date")) + # get sheet_names filter from config + sheet_names_filter = config.get("sheet_names", []) + # perform sheet's sync and get sheet's metadata and sheet loaded records for "sheet_metadata" and "sheets_loaded" streams sheet_metadata_records, sheets_loaded_records = sheets_load_data.load_data(catalog=catalog, state=state, selected_streams=selected_streams, sheets=sheets, - spreadsheet_time_extracted=time_extracted) + spreadsheet_time_extracted=time_extracted, + sheet_names_filter=sheet_names_filter) # sync "sheet_metadata" and "sheets_loaded" based on the records from spreadsheet metadata elif stream_name in ["sheet_metadata", "sheets_loaded"] and stream_name in selected_streams: diff --git a/tests/unittests/test_service_account_auth.py b/tests/unittests/test_service_account_auth.py new file mode 100644 index 0000000..600de23 --- /dev/null +++ b/tests/unittests/test_service_account_auth.py @@ -0,0 +1,307 @@ +import unittest +import json +from unittest import mock +from datetime import datetime, timedelta + +from tap_google_sheets.client import GoogleClient, SCOPES +from tap_google_sheets import validate_auth_config + + +class MockCredentials: + """Mock service account credentials for testing.""" + def __init__(self): + self.token = 'mock_sa_access_token' + self.expiry = datetime.utcnow() + timedelta(hours=1) + + def refresh(self, request): + self.token = 'refreshed_mock_sa_access_token' + self.expiry = datetime.utcnow() + timedelta(hours=1) + + +class MockResponse: + """Mock response object for requests calls.""" + def __init__(self, resp, status_code, content=[""], headers=None): + self.json_data = resp + self.status_code = status_code + self.content = content + self.headers = headers + + def json(self, object_pairs_hook=None): + return self.json_data + + +class TestConfigValidation(unittest.TestCase): + """Test authentication configuration validation.""" + + def test_valid_oauth2_config(self): + """Test that valid OAuth2 config returns 'oauth2'.""" + config = { + 'client_id': 'test_client_id', + 'client_secret': 'test_client_secret', + 'refresh_token': 'test_refresh_token', + 'spreadsheet_id': 'test_spreadsheet_id', + 'start_date': '2024-01-01T00:00:00Z', + 'user_agent': 'test-agent' + } + auth_type = validate_auth_config(config) + self.assertEqual(auth_type, 'oauth2') + + def test_valid_service_account_file_config(self): + """Test that valid service account file config returns 'service_account'.""" + config = { + 'credentials_file': '/path/to/key.json', + 'spreadsheet_id': 'test_spreadsheet_id', + 'start_date': '2024-01-01T00:00:00Z', + 'user_agent': 'test-agent' + } + auth_type = validate_auth_config(config) + self.assertEqual(auth_type, 'service_account') + + def test_valid_service_account_json_config(self): + """Test that valid service account JSON config returns 'service_account'.""" + config = { + 'credentials_json': '{"type": "service_account", "project_id": "test"}', + 'spreadsheet_id': 'test_spreadsheet_id', + 'start_date': '2024-01-01T00:00:00Z', + 'user_agent': 'test-agent' + } + auth_type = validate_auth_config(config) + self.assertEqual(auth_type, 'service_account') + + def test_error_both_oauth2_and_service_account(self): + """Test that providing both OAuth2 and service account credentials raises error.""" + config = { + 'client_id': 'test_client_id', + 'client_secret': 'test_client_secret', + 'refresh_token': 'test_refresh_token', + 'credentials_file': '/path/to/key.json', + 'spreadsheet_id': 'test_spreadsheet_id', + 'start_date': '2024-01-01T00:00:00Z', + 'user_agent': 'test-agent' + } + with self.assertRaises(Exception) as context: + validate_auth_config(config) + self.assertIn('Cannot specify both OAuth2 credentials', str(context.exception)) + + def test_error_neither_oauth2_nor_service_account(self): + """Test that providing neither OAuth2 nor service account credentials raises error.""" + config = { + 'spreadsheet_id': 'test_spreadsheet_id', + 'start_date': '2024-01-01T00:00:00Z', + 'user_agent': 'test-agent' + } + with self.assertRaises(Exception) as context: + validate_auth_config(config) + self.assertIn('Must specify either OAuth2 credentials', str(context.exception)) + + def test_error_both_credentials_file_and_json(self): + """Test that providing both credentials_file and credentials_json raises error.""" + config = { + 'credentials_file': '/path/to/key.json', + 'credentials_json': '{"type": "service_account"}', + 'spreadsheet_id': 'test_spreadsheet_id', + 'start_date': '2024-01-01T00:00:00Z', + 'user_agent': 'test-agent' + } + with self.assertRaises(Exception) as context: + validate_auth_config(config) + self.assertIn('Cannot specify both credentials_file and credentials_json', str(context.exception)) + + def test_empty_credentials_file_is_ignored(self): + """Test that empty credentials_file falls back to requiring OAuth2.""" + config = { + 'credentials_file': '', + 'spreadsheet_id': 'test_spreadsheet_id', + 'start_date': '2024-01-01T00:00:00Z', + 'user_agent': 'test-agent' + } + with self.assertRaises(Exception) as context: + validate_auth_config(config) + self.assertIn('Must specify either OAuth2 credentials', str(context.exception)) + + +class TestGoogleClientServiceAccount(unittest.TestCase): + """Test GoogleClient with service account authentication.""" + + @mock.patch('tap_google_sheets.client.service_account.Credentials.from_service_account_file') + def test_load_credentials_from_file(self, mock_from_file): + """Test loading service account credentials from file.""" + mock_credentials = MockCredentials() + mock_from_file.return_value = mock_credentials + + client = GoogleClient( + credentials_file='/path/to/key.json', + user_agent='test-agent' + ) + + # Trigger credential loading + client._load_service_account_credentials() + + mock_from_file.assert_called_once_with( + '/path/to/key.json', + scopes=SCOPES + ) + + @mock.patch('tap_google_sheets.client.service_account.Credentials.from_service_account_info') + def test_load_credentials_from_json(self, mock_from_info): + """Test loading service account credentials from inline JSON.""" + mock_credentials = MockCredentials() + mock_from_info.return_value = mock_credentials + + credentials_json = json.dumps({ + 'type': 'service_account', + 'project_id': 'test-project', + 'private_key_id': 'key-id', + 'private_key': 'fake-key', + 'client_email': 'sa@test.iam.gserviceaccount.com', + 'client_id': '123456789' + }) + + client = GoogleClient( + credentials_json=credentials_json, + user_agent='test-agent' + ) + + # Trigger credential loading + client._load_service_account_credentials() + + mock_from_info.assert_called_once() + call_args = mock_from_info.call_args + self.assertEqual(call_args[1]['scopes'], SCOPES) + self.assertEqual(call_args[0][0]['type'], 'service_account') + + @mock.patch('tap_google_sheets.client.GoogleAuthRequest') + @mock.patch('tap_google_sheets.client.service_account.Credentials.from_service_account_file') + def test_service_account_token_refresh(self, mock_from_file, mock_auth_request): + """Test that service account token is refreshed when needed.""" + mock_credentials = mock.MagicMock() + mock_credentials.token = None # Token not yet obtained + mock_credentials.expiry = None + mock_from_file.return_value = mock_credentials + + # Set up refresh to populate token + def do_refresh(request): + mock_credentials.token = 'refreshed_token' + mock_credentials.expiry = datetime.utcnow() + timedelta(hours=1) + mock_credentials.refresh.side_effect = do_refresh + + client = GoogleClient( + credentials_file='/path/to/key.json', + user_agent='test-agent' + ) + + client._get_service_account_token() + + # Verify refresh was called + mock_credentials.refresh.assert_called_once() + + @mock.patch('tap_google_sheets.client.requests.Session.request') + @mock.patch('tap_google_sheets.client.GoogleAuthRequest') + @mock.patch('tap_google_sheets.client.service_account.Credentials.from_service_account_file') + def test_request_uses_service_account_bearer_token(self, mock_from_file, mock_auth_request, mock_session_request): + """Test that API requests use the service account Bearer token.""" + mock_credentials = MockCredentials() + mock_credentials.token = 'sa_bearer_token_123' + mock_from_file.return_value = mock_credentials + + mock_session_request.return_value = MockResponse({'data': 'test'}, 200) + + client = GoogleClient( + credentials_file='/path/to/key.json', + user_agent='test-agent' + ) + + # Get access token first + client.get_access_token() + + # Make a request + client.request('GET', 'test/path') + + # Verify Bearer token was used in request + call_kwargs = mock_session_request.call_args[1] + self.assertEqual( + call_kwargs['headers']['Authorization'], + 'Bearer sa_bearer_token_123' + ) + + def test_auth_type_detection_service_account_file(self): + """Test that auth type is correctly detected for credentials_file.""" + client = GoogleClient( + credentials_file='/path/to/key.json', + user_agent='test-agent' + ) + self.assertEqual(client._GoogleClient__auth_type, 'service_account') + + def test_auth_type_detection_service_account_json(self): + """Test that auth type is correctly detected for credentials_json.""" + client = GoogleClient( + credentials_json='{"type": "service_account"}', + user_agent='test-agent' + ) + self.assertEqual(client._GoogleClient__auth_type, 'service_account') + + def test_auth_type_detection_oauth2(self): + """Test that auth type is correctly detected for OAuth2.""" + client = GoogleClient( + client_id='test_client_id', + client_secret='test_client_secret', + refresh_token='test_refresh_token', + user_agent='test-agent' + ) + self.assertEqual(client._GoogleClient__auth_type, 'oauth2') + + +class TestGoogleClientOAuth2Backward(unittest.TestCase): + """Test backward compatibility with OAuth2 authentication.""" + + @mock.patch('tap_google_sheets.client.requests.Session.post') + def test_oauth2_still_works(self, mock_post): + """Test that OAuth2 authentication still works as before.""" + mock_post.return_value = MockResponse({ + 'access_token': 'oauth2_token_123', + 'expires_in': 3600 + }, 200) + + client = GoogleClient( + client_id='test_client_id', + client_secret='test_client_secret', + refresh_token='test_refresh_token', + user_agent='test-agent' + ) + + client.get_access_token() + + # Verify OAuth2 endpoint was called + call_args = mock_post.call_args + self.assertIn('oauth2.googleapis.com/token', call_args[1]['url']) + + @mock.patch('tap_google_sheets.client.requests.Session.request') + @mock.patch('tap_google_sheets.client.requests.Session.post') + def test_oauth2_request_uses_correct_token(self, mock_post, mock_session_request): + """Test that OAuth2 requests use the correct Bearer token.""" + mock_post.return_value = MockResponse({ + 'access_token': 'oauth2_token_456', + 'expires_in': 3600 + }, 200) + mock_session_request.return_value = MockResponse({'data': 'test'}, 200) + + client = GoogleClient( + client_id='test_client_id', + client_secret='test_client_secret', + refresh_token='test_refresh_token', + user_agent='test-agent' + ) + + client.get_access_token() + client.request('GET', 'test/path') + + # Verify Bearer token was used + call_kwargs = mock_session_request.call_args[1] + self.assertEqual( + call_kwargs['headers']['Authorization'], + 'Bearer oauth2_token_456' + ) + + +if __name__ == '__main__': + unittest.main()